2024-04-22T21:28:23.829 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T21:28:23.829 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --name teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763 -- /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763/orig.config.yaml 2024-04-22T21:28:23.860 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T21:28:23.948 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7668763' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f workunit: branch: squid sha1: c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 207 sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f sleep_before_teardown: 0 subset: 24/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9 targets: smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO4HzJW48ghDW1vTuINRaToOKaci8aNMe27peObMwgOlmj3xo1tOOP5B6v0WwbEHJfuKk/FxMHMkEC8MMDkNwA4= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEeQqi5NdsyclP/NHZD0RJslPz89iyER8LjpFHca7zg7u1GcO7Auqf0dAsT3PISN4PZNZXvAmDgFsr91GsIac74= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T21:28:23.948 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa; will attempt to use it 2024-04-22T21:28:23.949 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks 2024-04-22T21:28:23.949 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T21:28:23.951 INFO:teuthology.task.internal:Checking packages... 2024-04-22T21:28:23.975 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '430e09df97c8fc7dc2b2ae424f68ed11366c540f' 2024-04-22T21:28:23.975 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T21:28:23.975 INFO:teuthology.packaging:ref: None 2024-04-22T21:28:23.975 INFO:teuthology.packaging:tag: None 2024-04-22T21:28:23.975 INFO:teuthology.packaging:branch: squid 2024-04-22T21:28:23.975 INFO:teuthology.packaging:sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:28:23.975 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-22T21:28:24.172 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2420.g246d7127 2024-04-22T21:28:24.173 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T21:28:24.179 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T21:28:24.180 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T21:28:24.186 INFO:teuthology.task.internal:Saving configuration 2024-04-22T21:28:24.211 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T21:28:24.217 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T21:28:24.244 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:23:14.315909', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO4HzJW48ghDW1vTuINRaToOKaci8aNMe27peObMwgOlmj3xo1tOOP5B6v0WwbEHJfuKk/FxMHMkEC8MMDkNwA4='} 2024-04-22T21:28:24.269 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:23:14.313914', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEeQqi5NdsyclP/NHZD0RJslPz89iyER8LjpFHca7zg7u1GcO7Auqf0dAsT3PISN4PZNZXvAmDgFsr91GsIac74='} 2024-04-22T21:28:24.269 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T21:28:24.275 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-22T21:28:24.276 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-22T21:28:24.276 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T21:28:24.350 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5247f360d0>, signals=[15]) 2024-04-22T21:28:24.351 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T21:28:24.356 INFO:teuthology.task.internal:Opening connections... 2024-04-22T21:28:24.356 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-04-22T21:28:24.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:28:24.434 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-04-22T21:28:24.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:28:24.512 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T21:28:24.519 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-04-22T21:28:24.537 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-04-22T21:28:24.537 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-04-22T21:28:24.595 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T21:28:24.596 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T21:28:24.597 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T21:28:24.597 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-04-22T21:28:24.618 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-04-22T21:28:24.636 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-04-22T21:28:24.636 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-04-22T21:28:24.693 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-04-22T21:28:24.693 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="9" 2024-04-22T21:28:24.693 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-04-22T21:28:24.693 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="9" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el9" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:LOGO="fedora-logo-icon" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T21:28:24.694 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T21:28:24.694 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-04-22T21:28:24.717 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T21:28:24.726 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T21:28:24.732 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T21:28:24.732 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-04-22T21:28:24.734 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-04-22T21:28:24.750 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T21:28:24.756 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T21:28:24.756 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-04-22T21:28:24.794 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-04-22T21:28:24.911 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T21:28:24.957 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T21:28:24.972 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T21:28:24.972 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T21:28:24.973 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T21:28:24.973 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T21:28:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:28:24.989 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-04-22T21:28:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:28:24.990 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-04-22T21:28:25.045 INFO:teuthology.orchestra.run.smithi188.stdout:5.14.0-437.el9.x86_64 2024-04-22T21:28:25.046 INFO:teuthology.task.kernel:Running kernel on smithi188: 5.14.0-437.el9.x86_64 2024-04-22T21:28:25.046 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2024-04-22T21:28:25.048 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-437.el9.x86_64 2024-04-22T21:28:25.048 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.14.0-437.el9.x86_64 2024-04-22T21:28:25.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum install -y kernel 2024-04-22T21:28:36.119 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 877 kB/s | 8.0 MB 00:09 2024-04-22T21:28:36.364 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 854 kB/s | 8.0 MB 00:09 2024-04-22T21:28:39.673 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 19 MB 00:02 2024-04-22T21:28:43.956 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.2 MB 00:00 2024-04-22T21:28:45.832 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-22T21:28:47.353 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-22T21:28:52.449 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-22T21:28:54.314 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T21:28:54.314 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T21:28:54.406 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-22T21:28:54.407 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-04-22T21:28:54.407 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-22T21:28:54.521 DEBUG:teuthology.orchestra.run.smithi165:> echo no | sudo yum reinstall kernel || true 2024-04-22T21:28:55.020 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 09:28:52 PM UTC. 2024-04-22T21:28:55.141 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-22T21:28:55.150 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-22T21:28:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-04-22T21:28:55.151 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-22T21:28:55.151 INFO:teuthology.orchestra.run.smithi165.stdout:Reinstalling: 2024-04-22T21:28:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:28:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:28:55.151 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-22T21:28:55.151 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-22T21:28:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:28:55.151 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 255 k 2024-04-22T21:28:55.152 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 0 2024-04-22T21:28:55.152 INFO:teuthology.orchestra.run.smithi165.stderr:Operation aborted. 2024-04-22T21:28:55.206 INFO:teuthology.orchestra.run.smithi165.stdout:Is this ok [y/N]: 2024-04-22T21:28:55.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum reinstall -y kernel || true 2024-04-22T21:28:55.701 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 09:28:52 PM UTC. 2024-04-22T21:28:55.819 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout:Reinstalling: 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:28:55.820 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 255 k 2024-04-22T21:28:55.821 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 0 2024-04-22T21:28:55.821 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-04-22T21:28:56.159 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-04-22T21:28:56.172 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-04-22T21:28:56.172 INFO:teuthology.orchestra.run.smithi165.stdout:Total 751 kB/s | 255 kB 00:00 2024-04-22T21:28:56.183 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-04-22T21:28:56.367 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-04-22T21:28:56.367 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-04-22T21:28:56.372 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-04-22T21:28:56.372 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-04-22T21:28:56.518 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-04-22T21:28:56.594 INFO:teuthology.orchestra.run.smithi165.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:28:56.595 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:28:57.259 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:28:57.259 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:28:57.939 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:28:57.939 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:28:57.939 INFO:teuthology.orchestra.run.smithi165.stdout:Reinstalled: 2024-04-22T21:28:57.939 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T21:28:57.939 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:28:57.939 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-04-22T21:28:58.142 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T21:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T21:28:58.189 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T21:28:58.189 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T21:28:58.189 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T21:28:58.189 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T21:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-04-22T21:28:58.248 DEBUG:teuthology.parallel:result is None 2024-04-22T21:28:59.356 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 878 kB/s | 19 MB 00:22 2024-04-22T21:29:03.598 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-22T21:29:05.216 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-04-22T21:29:07.002 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-22T21:29:12.016 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-22T21:29:13.906 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T21:29:13.906 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T21:29:13.998 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-22T21:29:13.999 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-22T21:29:14.000 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-22T21:29:14.116 DEBUG:teuthology.orchestra.run.smithi188:> echo no | sudo yum reinstall kernel || true 2024-04-22T21:29:14.616 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 09:29:12 PM UTC. 2024-04-22T21:29:14.736 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout:Reinstalling: 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-22T21:29:14.747 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 255 k 2024-04-22T21:29:14.748 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 0 2024-04-22T21:29:14.748 INFO:teuthology.orchestra.run.smithi188.stderr:Operation aborted. 2024-04-22T21:29:14.807 INFO:teuthology.orchestra.run.smithi188.stdout:Is this ok [y/N]: 2024-04-22T21:29:14.807 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum reinstall -y kernel || true 2024-04-22T21:29:15.305 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 09:29:12 PM UTC. 2024-04-22T21:29:15.421 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-22T21:29:15.422 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-22T21:29:15.422 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-22T21:29:15.422 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-22T21:29:15.422 INFO:teuthology.orchestra.run.smithi188.stdout:Reinstalling: 2024-04-22T21:29:15.422 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:29:15.423 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-22T21:29:15.423 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-22T21:29:15.423 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-22T21:29:15.423 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-22T21:29:15.423 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 255 k 2024-04-22T21:29:15.423 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 0 2024-04-22T21:29:15.423 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-22T21:29:15.743 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.4 MB/s | 255 kB 00:00 2024-04-22T21:29:15.743 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-22T21:29:15.743 INFO:teuthology.orchestra.run.smithi188.stdout:Total 795 kB/s | 255 kB 00:00 2024-04-22T21:29:15.760 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-22T21:29:15.966 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-22T21:29:15.966 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-22T21:29:15.971 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-22T21:29:15.971 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-22T21:29:16.118 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-22T21:29:16.202 INFO:teuthology.orchestra.run.smithi188.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:29:16.202 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:29:16.856 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:29:16.856 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:29:17.530 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:29:17.530 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-22T21:29:17.531 INFO:teuthology.orchestra.run.smithi188.stdout:Reinstalled: 2024-04-22T21:29:17.531 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T21:29:17.531 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-22T21:29:17.531 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-22T21:29:17.719 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T21:29:17.766 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T21:29:17.766 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T21:29:17.766 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T21:29:17.766 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-22T21:29:17.766 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T21:29:17.794 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-04-22T21:29:17.822 DEBUG:teuthology.parallel:result is None 2024-04-22T21:29:17.822 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T21:29:17.829 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T21:29:17.830 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T21:29:17.832 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T21:29:17.877 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T21:29:17.883 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T21:29:17.892 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T21:29:17.892 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T21:29:17.895 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T21:29:17.942 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T21:29:17.952 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T21:29:17.952 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T21:29:17.960 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T21:29:17.995 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:29:18.005 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:29:18.010 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:29:18.019 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:29:18.021 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T21:29:18.028 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T21:29:18.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T21:29:18.049 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T21:29:18.084 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T21:29:18.093 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T21:29:18.094 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T21:29:18.115 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T21:29:18.139 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:29:18.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:29:18.259 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:29:18.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:29:18.382 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:29:18.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T21:29:18.446 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:29:18.488 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:29:18.536 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:29:18.620 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:29:18.656 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-22T21:29:18.656 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T21:29:18.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-04-22T21:29:18.721 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-04-22T21:29:18.758 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:29:18.801 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:29:19.298 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T21:29:19.305 INFO:teuthology.task.internal:Starting timer... 2024-04-22T21:29:19.306 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T21:29:19.322 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T21:29:19.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-04-22T21:29:19.379 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-04-22T21:29:19.381 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-04-22T21:29:19.442 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-04-22T21:29:19.444 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T21:29:19.444 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-04-22T21:29:19.480 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-04-22T21:29:19.480 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-04-22T21:29:19.508 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-04-22T21:29:19.508 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi165.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-04-22T21:29:19.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T21:29:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:29:19.545 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T21:29:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:29:19.572 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T21:29:19.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-04-22T21:29:19.609 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-04-22T21:29:19.637 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T21:29:19.646 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T21:29:19.664 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T21:29:19.665 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi165.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T21:32:46.741 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi165.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-04-22T21:32:46.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-04-22T21:32:46.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:32:46.817 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-04-22T21:32:46.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-04-22T21:32:46.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-22T21:32:46.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:32:46.974 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-22T21:32:47.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-22T21:32:47.018 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T21:32:47.029 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T21:32:47.029 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T21:32:47.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T21:32:47.033 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T21:32:47.033 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T21:32:47.057 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T21:32:47.071 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T21:32:47.085 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T21:32:47.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-04-22T21:32:47.098 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T21:32:47.103 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-04-22T21:32:47.116 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T21:32:47.120 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-04-22T21:32:47.129 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T21:32:47.130 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-04-22T21:32:47.143 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T21:32:47.156 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T21:32:47.185 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-04-22T21:32:47.188 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T21:32:47.188 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-04-22T21:32:47.188 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:32:47.188 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:32:47.188 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:32:47.188 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:32:47.220 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-04-22T21:32:47.224 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T21:32:47.224 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-22T21:32:47.224 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:32:47.224 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:32:47.224 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:32:47.224 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:32:47.224 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T21:32:47.324 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '430e09df97c8fc7dc2b2ae424f68ed11366c540f'} 2024-04-22T21:32:47.324 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-22T21:32:47.325 INFO:tasks.cephadm:Cluster fsid is dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:32:47.325 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T21:32:47.325 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.165', 'mon.c': '[v2:172.21.15.165:3301,v1:172.21.15.165:6790]', 'mon.b': '172.21.15.188'} 2024-04-22T21:32:47.325 INFO:tasks.cephadm:First mon is mon.a on smithi165 2024-04-22T21:32:47.325 INFO:tasks.cephadm:First mgr is y 2024-04-22T21:32:47.325 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T21:32:47.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-04-22T21:32:47.350 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2024-04-22T21:32:47.375 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-22T21:32:47.375 DEBUG:teuthology.orchestra.run.smithi165:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T21:32:47.669 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 22 21:32 /home/ubuntu/cephtest/cephadm 2024-04-22T21:32:47.670 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T21:32:47.836 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 22 21:32 /home/ubuntu/cephtest/cephadm 2024-04-22T21:32:47.837 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T21:32:47.852 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T21:32:47.877 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-22T21:32:47.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-22T21:32:47.895 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-22T21:32:49.426 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T21:32:49.445 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T21:33:07.843 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-22T21:33:07.844 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-22T21:33:07.844 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-22T21:33:07.844 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2024-04-22T21:33:07.844 INFO:teuthology.orchestra.run.smithi188.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-22T21:33:07.844 INFO:teuthology.orchestra.run.smithi188.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-22T21:33:07.844 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-22T21:33:07.844 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-22T21:33:13.178 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-04-22T21:33:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-22T21:33:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-22T21:33:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-04-22T21:33:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-22T21:33:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-22T21:33:13.178 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-04-22T21:33:13.178 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-04-22T21:33:13.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-04-22T21:33:13.248 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2024-04-22T21:33:13.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-04-22T21:33:13.302 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2024-04-22T21:33:13.347 INFO:tasks.cephadm:Writing seed config... 2024-04-22T21:33:13.348 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-22T21:33:13.348 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T21:33:13.348 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T21:33:13.348 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-04-22T21:33:13.348 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T21:33:13.348 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T21:33:13.348 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T21:33:13.348 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T21:33:13.348 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T21:33:13.349 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:33:13.349 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T21:33:13.365 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = dccb09a4-00ef-11ef-bc93-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-22T21:33:13.365 DEBUG:teuthology.orchestra.run.smithi165:mon.a> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.a.service 2024-04-22T21:33:13.408 DEBUG:teuthology.orchestra.run.smithi165:mgr.y> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mgr.y.service 2024-04-22T21:33:13.450 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T21:33:13.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.165 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:33:13.641 INFO:teuthology.orchestra.run.smithi165.stderr:-------------------------------------------------------------------------------- 2024-04-22T21:33:13.642 INFO:teuthology.orchestra.run.smithi165.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'dccb09a4-00ef-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.165', '--skip-admin-label'] 2024-04-22T21:33:13.665 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: 4.9.4-dev 2024-04-22T21:33:13.667 INFO:teuthology.orchestra.run.smithi165.stderr:Verifying podman|docker is present... 2024-04-22T21:33:13.694 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: 4.9.4-dev 2024-04-22T21:33:13.696 INFO:teuthology.orchestra.run.smithi165.stderr:Verifying lvm2 is present... 2024-04-22T21:33:13.696 INFO:teuthology.orchestra.run.smithi165.stderr:Verifying time synchronization is in place... 2024-04-22T21:33:13.703 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T21:33:13.709 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: inactive 2024-04-22T21:33:13.716 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: enabled 2024-04-22T21:33:13.722 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: active 2024-04-22T21:33:13.722 INFO:teuthology.orchestra.run.smithi165.stderr:Unit chronyd.service is enabled and running 2024-04-22T21:33:13.722 INFO:teuthology.orchestra.run.smithi165.stderr:Repeating the final host check... 2024-04-22T21:33:13.744 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: 4.9.4-dev 2024-04-22T21:33:13.746 INFO:teuthology.orchestra.run.smithi165.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-22T21:33:13.746 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl is present 2024-04-22T21:33:13.746 INFO:teuthology.orchestra.run.smithi165.stderr:lvcreate is present 2024-04-22T21:33:13.752 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T21:33:13.758 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: inactive 2024-04-22T21:33:13.764 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: enabled 2024-04-22T21:33:13.770 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: active 2024-04-22T21:33:13.771 INFO:teuthology.orchestra.run.smithi165.stderr:Unit chronyd.service is enabled and running 2024-04-22T21:33:13.771 INFO:teuthology.orchestra.run.smithi165.stderr:Host looks OK 2024-04-22T21:33:13.771 INFO:teuthology.orchestra.run.smithi165.stderr:Cluster fsid: dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:13.771 INFO:teuthology.orchestra.run.smithi165.stderr:Acquiring lock 140208375156592 on /run/cephadm/dccb09a4-00ef-11ef-bc93-c7b262605968.lock 2024-04-22T21:33:13.771 INFO:teuthology.orchestra.run.smithi165.stderr:Lock 140208375156592 acquired on /run/cephadm/dccb09a4-00ef-11ef-bc93-c7b262605968.lock 2024-04-22T21:33:13.772 INFO:teuthology.orchestra.run.smithi165.stderr:Verifying IP 172.21.15.165 port 3300 ... 2024-04-22T21:33:13.772 INFO:teuthology.orchestra.run.smithi165.stderr:Verifying IP 172.21.15.165 port 6789 ... 2024-04-22T21:33:13.772 INFO:teuthology.orchestra.run.smithi165.stderr:Base mon IP is 172.21.15.165, final addrv is [v2:172.21.15.165:3300,v1:172.21.15.165:6789] 2024-04-22T21:33:13.774 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.165 metric 100 2024-04-22T21:33:13.774 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.165 metric 100 2024-04-22T21:33:13.777 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T21:33:13.777 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-22T21:33:13.777 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-22T21:33:13.780 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T21:33:13.780 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-22T21:33:13.780 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-22T21:33:13.780 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-22T21:33:13.780 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: inet6 fe80::c491:e4b2:29d1:c2cd/64 scope link noprefixroute 2024-04-22T21:33:13.780 INFO:teuthology.orchestra.run.smithi165.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-22T21:33:13.780 INFO:teuthology.orchestra.run.smithi165.stderr:Mon IP `172.21.15.165` is in CIDR network `172.21.0.0/20` 2024-04-22T21:33:13.781 INFO:teuthology.orchestra.run.smithi165.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T21:33:13.781 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T21:33:13.882 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-22T21:33:14.266 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: Getting image source signatures 2024-04-22T21:33:14.266 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-22T21:33:14.266 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-22T21:33:14.266 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-22T21:33:14.266 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-22T21:33:14.266 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-22T21:33:14.266 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-22T21:33:14.439 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: Writing manifest to image destination 2024-04-22T21:33:14.591 INFO:teuthology.orchestra.run.smithi165.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-22T21:33:14.970 INFO:teuthology.orchestra.run.smithi165.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-22T21:33:15.506 INFO:teuthology.orchestra.run.smithi165.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-22T21:33:15.507 INFO:teuthology.orchestra.run.smithi165.stderr:Extracting ceph user uid/gid from container image... 2024-04-22T21:33:15.981 INFO:teuthology.orchestra.run.smithi165.stderr:stat: 167 167 2024-04-22T21:33:16.479 INFO:teuthology.orchestra.run.smithi165.stderr:Creating initial keys... 2024-04-22T21:33:16.894 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph-authtool: AQCc1yZmgkhNMRAAbCin3lycHosl+EEmxw2D0g== 2024-04-22T21:33:17.883 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph-authtool: AQCd1yZmfwkkMxAAu1ZHIHULf0fwIYHg1gh1PQ== 2024-04-22T21:33:18.837 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph-authtool: AQCe1yZm3xV7LRAA0usipsyppdK5xNkNUfokHg== 2024-04-22T21:33:19.470 INFO:teuthology.orchestra.run.smithi165.stderr:Creating initial monmap... 2024-04-22T21:33:19.935 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T21:33:19.935 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-22T21:33:19.935 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:19.935 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T21:33:20.466 INFO:teuthology.orchestra.run.smithi165.stderr:monmaptool for a [v2:172.21.15.165:3300,v1:172.21.15.165:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T21:33:20.466 INFO:teuthology.orchestra.run.smithi165.stderr:setting min_mon_release = octopus 2024-04-22T21:33:20.466 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/monmaptool: set fsid to dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:20.466 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T21:33:20.466 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:33:20.466 INFO:teuthology.orchestra.run.smithi165.stderr:Creating mon... 2024-04-22T21:33:21.429 INFO:teuthology.orchestra.run.smithi165.stderr:create mon.a on 2024-04-22T21:33:21.648 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T21:33:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968.target. 2024-04-22T21:33:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968.target. 2024-04-22T21:33:22.130 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: Failed to reset failed state of unit ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-22T21:33:22.138 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968.target.wants/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@.service. 2024-04-22T21:33:22.570 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:22 smithi165 podman[21468]: 2024-04-22 21:33:22.588025656 +0000 UTC m=+0.119406947 container create e705fa793c6b573a880316b680d40c3543fde556bc0e2ee01ef4bea1113a97f8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., version=8, release=754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , distribution-scope=public) 2024-04-22T21:33:22.822 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:22 smithi165 podman[21468]: 2024-04-22 21:33:22.790122533 +0000 UTC m=+0.321503824 container init e705fa793c6b573a880316b680d40c3543fde556bc0e2ee01ef4bea1113a97f8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, release=754, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, ceph=True, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T21:33:22.822 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:22 smithi165 podman[21468]: 2024-04-22 21:33:22.793545202 +0000 UTC m=+0.324926490 container start e705fa793c6b573a880316b680d40c3543fde556bc0e2ee01ef4bea1113a97f8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-type=git, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, release=754, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True) 2024-04-22T21:33:22.822 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:22 smithi165 ceph-mon[21486]: mkfs dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:22.822 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:22 smithi165 ceph-mon[21486]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T21:33:22.878 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: disabled 2024-04-22T21:33:22.885 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: inactive 2024-04-22T21:33:22.886 INFO:teuthology.orchestra.run.smithi165.stderr:firewalld.service is not enabled 2024-04-22T21:33:22.886 INFO:teuthology.orchestra.run.smithi165.stderr:Not possible to enable service . firewalld.service is not available 2024-04-22T21:33:22.887 INFO:teuthology.orchestra.run.smithi165.stderr:Waiting for mon to start... 2024-04-22T21:33:22.887 INFO:teuthology.orchestra.run.smithi165.stderr:Waiting for mon... 2024-04-22T21:33:23.104 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:22 smithi165 bash[21468]: e705fa793c6b573a880316b680d40c3543fde556bc0e2ee01ef4bea1113a97f8 2024-04-22T21:33:23.104 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:22 smithi165 systemd[1]: Started Ceph mon.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: cluster: 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: id: dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: services: 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.578617s) 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: data: 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-22T21:33:23.397 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: pgs: 2024-04-22T21:33:23.398 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:23.970 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:23 smithi165 ceph-mon[21486]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T21:33:23.970 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:23 smithi165 ceph-mon[21486]: monmap e1: 1 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} 2024-04-22T21:33:23.970 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:23 smithi165 ceph-mon[21486]: fsmap 2024-04-22T21:33:23.970 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:23 smithi165 ceph-mon[21486]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T21:33:23.970 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:23 smithi165 ceph-mon[21486]: mgrmap e1: no daemons active 2024-04-22T21:33:23.971 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:23 smithi165 ceph-mon[21486]: from='client.? 172.21.15.165:0/1763901825' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T21:33:23.972 INFO:teuthology.orchestra.run.smithi165.stderr:mon is available 2024-04-22T21:33:23.973 INFO:teuthology.orchestra.run.smithi165.stderr:Assimilating anything we can from ceph.conf... 2024-04-22T21:33:24.420 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: [global] 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: fsid = dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.165:3300,v1:172.21.15.165:6789] 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: [mgr] 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-22T21:33:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:24.422 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: [osd] 2024-04-22T21:33:24.422 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-22T21:33:24.422 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-22T21:33:24.937 INFO:teuthology.orchestra.run.smithi165.stderr:Generating new minimal ceph.conf... 2024-04-22T21:33:25.035 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:24 smithi165 ceph-mon[21486]: from='client.? 172.21.15.165:0/3237110191' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T21:33:25.035 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:24 smithi165 ceph-mon[21486]: from='client.? 172.21.15.165:0/3237110191' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T21:33:25.917 INFO:teuthology.orchestra.run.smithi165.stderr:Restarting the monitor... 2024-04-22T21:33:25.935 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:25 smithi165 ceph-mon[21486]: from='client.? 172.21.15.165:0/1132043838' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:33:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:25 smithi165 systemd[1]: Stopping Ceph mon.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:33:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:25 smithi165 bash[21707]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-22T21:33:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a[21482]: 2024-04-22T21:33:26.093+0000 7ff3a628a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:33:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a[21482]: 2024-04-22T21:33:26.093+0000 7ff3a628a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T21:33:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 podman[21718]: 2024-04-22 21:33:26.11440583 +0000 UTC m=+0.125949184 container died e705fa793c6b573a880316b680d40c3543fde556bc0e2ee01ef4bea1113a97f8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, version=8, RELEASE=HEAD, distribution-scope=public, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T21:33:26.630 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 podman[21718]: 2024-04-22 21:33:26.445048944 +0000 UTC m=+0.456592279 container cleanup e705fa793c6b573a880316b680d40c3543fde556bc0e2ee01ef4bea1113a97f8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, name=centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, version=8, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-04-22T21:33:26.630 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 bash[21718]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a 2024-04-22T21:33:26.630 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 podman[21732]: 2024-04-22 21:33:26.629121593 +0000 UTC m=+0.512943386 container remove e705fa793c6b573a880316b680d40c3543fde556bc0e2ee01ef4bea1113a97f8 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, distribution-scope=public, vcs-type=git, RELEASE=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.openshift.tags=base centos centos-stream) 2024-04-22T21:33:26.630 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 bash[21744]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-22T21:33:26.895 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-22T21:33:26.896 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 systemd[1]: Stopped Ceph mon.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:33:26.896 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 systemd[1]: Starting Ceph mon.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:33:26.896 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 podman[21813]: 2024-04-22 21:33:26.794353799 +0000 UTC m=+0.018530406 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-22T21:33:26.896 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:26 smithi165 podman[21813]: 2024-04-22 21:33:26.913184709 +0000 UTC m=+0.137361364 container create b2377729385b94128255e6ac38bcefaba4425e89cc321f74a1a3cb04c0684e51 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, release=754, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, ceph=True, io.openshift.expose-services=, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux ) 2024-04-22T21:33:27.309 INFO:teuthology.orchestra.run.smithi165.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-22T21:33:27.508 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 podman[21813]: 2024-04-22 21:33:27.265243018 +0000 UTC m=+0.489419611 container init b2377729385b94128255e6ac38bcefaba4425e89cc321f74a1a3cb04c0684e51 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, vcs-type=git, version=8, RELEASE=HEAD, GIT_CLEAN=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T21:33:27.508 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 podman[21813]: 2024-04-22 21:33:27.268648735 +0000 UTC m=+0.492825335 container start b2377729385b94128255e6ac38bcefaba4425e89cc321f74a1a3cb04c0684e51 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, ceph=True, version=8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, architecture=x86_64, release=754, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream) 2024-04-22T21:33:27.508 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T21:33:27.508 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-22T21:33:27.508 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: pidfile_write: ignore empty --pid-file 2024-04-22T21:33:27.509 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: load: jerasure load: lrc 2024-04-22T21:33:27.509 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: RocksDB version: 6.15.5 2024-04-22T21:33:27.509 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Compile date Apr 18 2022 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: DB SUMMARY 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: DB Session ID: DOVY0JG342DQALPLR1NU 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: CURRENT file: CURRENT 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82663 ; 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.error_if_exists: 0 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.create_if_missing: 0 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.paranoid_checks: 1 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.env: 0x557ed72c6860 2024-04-22T21:33:27.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.fs: Posix File System 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.info_log: 0x557ed8febdc0 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.statistics: (nil) 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.use_fsync: 0 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_log_file_size: 0 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.allow_fallocate: 1 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.use_direct_reads: 0 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T21:33:27.511 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.db_log_dir: 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.write_buffer_manager: 0x557ed90dc240 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T21:33:27.512 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.unordered_write: 0 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.row_cache: None 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.wal_filter: None 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T21:33:27.513 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.preserve_deletes: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.two_write_queues: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.atomic_flush: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.log_readahead_size: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_background_jobs: 2 2024-04-22T21:33:27.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_background_compactions: -1 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_subcompactions: 1 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_open_files: -1 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T21:33:27.515 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_background_flushes: -1 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Compression algorithms supported: 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: kZSTD supported: 0 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: kXpressCompression supported: 0 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: kLZ4Compression supported: 1 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: kBZip2Compression supported: 0 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: kZlibCompression supported: 1 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: kSnappyCompression supported: 1 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.merge_operator: 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_filter: None 2024-04-22T21:33:27.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557ed8fb7d00) 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: cache_index_and_filter_blocks: 1 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: pin_top_level_index_and_filter: 1 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: index_type: 0 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: data_block_index_type: 0 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: index_shortening: 1 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: hash_index_allow_collision: 1 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: checksum: 1 2024-04-22T21:33:27.517 INFO:journalctl@ceph.mon.a.smithi165.stdout: no_block_cache: 0 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: block_cache: 0x557ed9022170 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: block_cache_name: BinnedLRUCache 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: block_cache_options: 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: capacity : 536870912 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: num_shard_bits : 4 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: strict_capacity_limit : 0 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: high_pri_pool_ratio: 0.000 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: block_cache_compressed: (nil) 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: persistent_cache: (nil) 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: block_size: 4096 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: block_size_deviation: 10 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: block_restart_interval: 16 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: index_block_restart_interval: 1 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: metadata_block_size: 4096 2024-04-22T21:33:27.518 INFO:journalctl@ceph.mon.a.smithi165.stdout: partition_filters: 0 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout: use_delta_encoding: 1 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout: whole_key_filtering: 1 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout: verify_compression: 0 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout: read_amp_bytes_per_bit: 0 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout: format_version: 4 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout: enable_index_compression: 1 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout: block_align: 0 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compression: NoCompression 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.num_levels: 7 2024-04-22T21:33:27.519 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T21:33:27.520 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T21:33:27.521 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T21:33:27.521 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T21:33:27.521 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T21:33:27.522 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T21:33:27.523 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.table_properties_collectors: 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.inplace_update_support: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.bloom_locality: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.max_successive_merges: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.ttl: 2592000 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.enable_blob_files: false 2024-04-22T21:33:27.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.min_blob_size: 0 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821607293982, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821607294681, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79213, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77476, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71785, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713821607, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "37726dda-e840-4844-ae5b-ecb137fd9427", "db_session_id": "DOVY0JG342DQALPLR1NU"}} 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821607295012, "job": 1, "event": "recovery_finished"} 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x557ed9008a80 2024-04-22T21:33:27.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: DB pointer 0x557ed9018000 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: ** DB Stats ** 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: ** Compaction Stats [default] ** 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:33:27.526 INFO:journalctl@ceph.mon.a.smithi165.stdout: L0 2/0 79.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.4 0.00 0.00 1 0.000 0 0 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: Sum 2/0 79.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.4 0.00 0.00 1 0.000 0 0 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.4 0.00 0.00 1 0.000 0 0 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: ** Compaction Stats [default] ** 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.4 0.00 0.00 1 0.000 0 0 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: Cumulative compaction: 0.00 GB write, 26.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:33:27.527 INFO:journalctl@ceph.mon.a.smithi165.stdout: Interval compaction: 0.00 GB write, 26.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: ** Compaction Stats [default] ** 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: L0 2/0 79.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.4 0.00 0.00 1 0.000 0 0 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: Sum 2/0 79.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.4 0.00 0.00 1 0.000 0 0 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: ** Compaction Stats [default] ** 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.4 0.00 0.00 1 0.000 0 0 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:33:27.528 INFO:journalctl@ceph.mon.a.smithi165.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: Cumulative compaction: 0.00 GB write, 25.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: starting mon.a rank 0 at public addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] at bind addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mon.a@-1(???) e1 preinit fsid dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mon.a@-1(???).mds e1 new map 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mon.a@-1(???).mds e1 print_map 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: e1 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:33:27.529 INFO:journalctl@ceph.mon.a.smithi165.stdout: legacy client fscid: -1 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout: 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout: No filesystems configured 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta expand map: {default=false} 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta from 'false' to 'false' 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta expanded map: {default=false} 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta expand map: {default=info} 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta from 'info' to 'info' 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta expanded map: {default=info} 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta expand map: {default=daemon} 2024-04-22T21:33:27.530 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta expanded map: {default=daemon} 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta expand map: {default=debug} 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta from 'debug' to 'debug' 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: expand_channel_meta expanded map: {default=debug} 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 bash[21813]: b2377729385b94128255e6ac38bcefaba4425e89cc321f74a1a3cb04c0684e51 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 systemd[1]: Started Ceph mon.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: monmap e1: 1 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: fsmap 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T21:33:27.531 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:27 smithi165 ceph-mon[21833]: mgrmap e1: no daemons active 2024-04-22T21:33:28.347 INFO:teuthology.orchestra.run.smithi165.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-22T21:33:28.348 INFO:teuthology.orchestra.run.smithi165.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:33:28.348 INFO:teuthology.orchestra.run.smithi165.stderr:Creating mgr... 2024-04-22T21:33:28.348 INFO:teuthology.orchestra.run.smithi165.stderr:Verifying port 9283 ... 2024-04-22T21:33:28.520 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: Failed to reset failed state of unit ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-22T21:33:28.527 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968.target.wants/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@.service. 2024-04-22T21:33:29.002 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:28 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/221704388' entity='client.admin' 2024-04-22T21:33:29.211 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: disabled 2024-04-22T21:33:29.219 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: inactive 2024-04-22T21:33:29.220 INFO:teuthology.orchestra.run.smithi165.stderr:firewalld.service is not enabled 2024-04-22T21:33:29.220 INFO:teuthology.orchestra.run.smithi165.stderr:Not possible to enable service . firewalld.service is not available 2024-04-22T21:33:29.228 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: disabled 2024-04-22T21:33:29.235 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: inactive 2024-04-22T21:33:29.236 INFO:teuthology.orchestra.run.smithi165.stderr:firewalld.service is not enabled 2024-04-22T21:33:29.236 INFO:teuthology.orchestra.run.smithi165.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-22T21:33:29.236 INFO:teuthology.orchestra.run.smithi165.stderr:Waiting for mgr to start... 2024-04-22T21:33:29.236 INFO:teuthology.orchestra.run.smithi165.stderr:Waiting for mgr... 2024-04-22T21:33:29.362 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:29 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:29.253+0000 7f5d9a116000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T21:33:29.659 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:29 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:29.386+0000 7f5d9a116000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:33:29.659 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:29 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:29.483+0000 7f5d9a116000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: { 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "fsid": "dccb09a4-00ef-11ef-bc93-c7b262605968", 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "health": { 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T21:33:29.764 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 0 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ], 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "a" 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ], 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T21:33:29.765 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T21:33:29.766 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T21:33:29.766 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T21:33:29.769 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "available": false, 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "iostat", 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "nfs", 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "restful" 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ], 2024-04-22T21:33:29.770 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:33:29.771 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:29.771 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T21:33:29.771 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:29.771 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "modified": "2024-04-22T21:33:22.832088+0000", 2024-04-22T21:33:29.771 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:33:29.771 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:29.771 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T21:33:29.771 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: } 2024-04-22T21:33:30.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:29 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3180921021' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T21:33:30.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:29 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:29.806+0000 7f5d9a116000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T21:33:30.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:29 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:29.888+0000 7f5d9a116000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T21:33:30.374 INFO:teuthology.orchestra.run.smithi165.stderr:mgr not available, waiting (1/15)... 2024-04-22T21:33:31.309 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:31 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:31.021+0000 7f5d9a116000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T21:33:31.309 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:31 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:31.184+0000 7f5d9a116000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T21:33:31.309 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:31 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:31.263+0000 7f5d9a116000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T21:33:31.809 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:31 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:31.573+0000 7f5d9a116000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T21:33:31.809 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:31 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:31.662+0000 7f5d9a116000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T21:33:31.809 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:31 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:31.753+0000 7f5d9a116000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T21:33:32.309 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:31 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:31.994+0000 7f5d9a116000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:33:32.809 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:32 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:32.438+0000 7f5d9a116000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T21:33:33.256 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:32 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:32.927+0000 7f5d9a116000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T21:33:33.257 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:33.020+0000 7f5d9a116000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T21:33:33.514 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:33.322+0000 7f5d9a116000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T21:33:33.514 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:33.412+0000 7f5d9a116000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T21:33:34.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:33.820+0000 7f5d9a116000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T21:33:34.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:33.954+0000 7f5d9a116000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T21:33:34.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:34.046+0000 7f5d9a116000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T21:33:34.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:34.160+0000 7f5d9a116000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T21:33:34.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:34.251+0000 7f5d9a116000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T21:33:35.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:34.816+0000 7f5d9a116000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T21:33:35.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:35 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:35.191+0000 7f5d9a116000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T21:33:35.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:35 smithi165 ceph-mon[21833]: Activating manager daemon y 2024-04-22T21:33:35.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:35 smithi165 ceph-mon[21833]: mgrmap e2: y(active, starting, since 0.0036798s) 2024-04-22T21:33:35.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:35 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:33:35.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:35 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:33:35.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:35 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:33:35.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:35 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:33:35.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:35 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T21:33:35.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:35 smithi165 ceph-mon[21833]: Manager daemon y is now available 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: { 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "fsid": "dccb09a4-00ef-11ef-bc93-c7b262605968", 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "health": { 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 0 2024-04-22T21:33:36.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ], 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "a" 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ], 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T21:33:36.103 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:36.104 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "available": false, 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "iostat", 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "nfs", 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "restful" 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ], 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "modified": "2024-04-22T21:33:22.832088+0000", 2024-04-22T21:33:36.105 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:33:36.106 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:36.106 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T21:33:36.106 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: } 2024-04-22T21:33:36.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:36 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:33:36.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:36 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:33:36.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:36 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' 2024-04-22T21:33:36.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:36 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' 2024-04-22T21:33:36.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:36 smithi165 ceph-mon[21833]: from='mgr.14100 172.21.15.165:0/393627806' entity='mgr.y' 2024-04-22T21:33:36.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:36 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2066897260' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T21:33:36.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:36 smithi165 ceph-mon[21833]: mgrmap e3: y(active, since 1.00659s) 2024-04-22T21:33:38.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:38 smithi165 ceph-mon[21833]: mgrmap e4: y(active, since 2s) 2024-04-22T21:33:41.250 INFO:teuthology.orchestra.run.smithi165.stderr:mgr not available, waiting (2/15)... 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: { 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "fsid": "dccb09a4-00ef-11ef-bc93-c7b262605968", 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "health": { 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 0 2024-04-22T21:33:43.901 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ], 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "a" 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ], 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "quorum_age": 16, 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T21:33:43.902 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T21:33:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "available": true, 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "iostat", 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "nfs", 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "restful" 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ], 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "modified": "2024-04-22T21:33:22.832088+0000", 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "services": {} 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: }, 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T21:33:43.904 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: } 2024-04-22T21:33:44.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:43 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/932114633' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T21:33:44.420 INFO:teuthology.orchestra.run.smithi165.stderr:mgr is available 2024-04-22T21:33:45.027 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:45.028 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: [global] 2024-04-22T21:33:45.028 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: fsid = dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:45.028 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-22T21:33:45.028 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-22T21:33:45.028 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-22T21:33:45.028 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-22T21:33:45.028 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:45.028 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: [mgr] 2024-04-22T21:33:45.028 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-22T21:33:45.029 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 2024-04-22T21:33:45.029 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: [osd] 2024-04-22T21:33:45.029 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-22T21:33:45.029 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-22T21:33:45.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:45 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2055253378' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T21:33:45.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:45 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2055253378' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T21:33:45.626 INFO:teuthology.orchestra.run.smithi165.stderr:Enabling cephadm module... 2024-04-22T21:33:46.601 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:46 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3556852439' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T21:33:46.602 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:46 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ignoring --setuser ceph since I am not root 2024-04-22T21:33:46.602 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:46 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ignoring --setgroup ceph since I am not root 2024-04-22T21:33:46.602 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:46 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:46.489+0000 7fd1886ec000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T21:33:46.602 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:46 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:46.617+0000 7fd1886ec000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:33:46.888 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:46 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:46.716+0000 7fd1886ec000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T21:33:47.214 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:47 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:47.029+0000 7fd1886ec000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T21:33:47.214 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:47 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:47.111+0000 7fd1886ec000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T21:33:47.468 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:47 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3556852439' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T21:33:47.468 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:47 smithi165 ceph-mon[21833]: mgrmap e5: y(active, since 11s) 2024-04-22T21:33:47.923 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: { 2024-04-22T21:33:47.923 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-22T21:33:47.923 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "available": true, 2024-04-22T21:33:47.923 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-22T21:33:47.923 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-22T21:33:47.924 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: } 2024-04-22T21:33:48.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:48 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1647520062' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T21:33:48.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:48 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:48.269+0000 7fd1886ec000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T21:33:48.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:48 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:48.435+0000 7fd1886ec000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T21:33:48.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:48 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:48.517+0000 7fd1886ec000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T21:33:48.672 INFO:teuthology.orchestra.run.smithi165.stderr:Waiting for the mgr to restart... 2024-04-22T21:33:48.672 INFO:teuthology.orchestra.run.smithi165.stderr:Waiting for mgr epoch 5... 2024-04-22T21:33:48.919 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:48 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:48.844+0000 7fd1886ec000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T21:33:48.919 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:48 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:48.935+0000 7fd1886ec000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T21:33:49.262 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:49 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:49.033+0000 7fd1886ec000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T21:33:49.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:49 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:49.278+0000 7fd1886ec000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:33:50.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:49 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:49.716+0000 7fd1886ec000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T21:33:50.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:50 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:50.198+0000 7fd1886ec000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T21:33:50.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:50 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:50.291+0000 7fd1886ec000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T21:33:51.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:50 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:50.583+0000 7fd1886ec000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T21:33:51.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:50 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:50.672+0000 7fd1886ec000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T21:33:51.423 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:51 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:51.101+0000 7fd1886ec000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T21:33:51.424 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:51 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:51.231+0000 7fd1886ec000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T21:33:51.424 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:51 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:51.323+0000 7fd1886ec000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T21:33:51.424 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:51 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:51.439+0000 7fd1886ec000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T21:33:51.809 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:51 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:51.529+0000 7fd1886ec000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T21:33:52.309 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:52 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:52.074+0000 7fd1886ec000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T21:33:52.708 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:52 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:33:52.469+0000 7fd1886ec000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T21:33:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:53 smithi165 ceph-mon[21833]: Active manager daemon y restarted 2024-04-22T21:33:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:53 smithi165 ceph-mon[21833]: Activating manager daemon y 2024-04-22T21:33:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:53 smithi165 ceph-mon[21833]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T21:33:54.302 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: { 2024-04-22T21:33:54.302 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-22T21:33:54.302 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "initialized": true 2024-04-22T21:33:54.302 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: } 2024-04-22T21:33:54.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:33:54] ENGINE Bus STARTING 2024-04-22T21:33:54.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:33:54] ENGINE Serving on https://172.21.15.165:7150 2024-04-22T21:33:54.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:33:54] ENGINE Bus STARTED 2024-04-22T21:33:54.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: mgrmap e6: y(active, starting, since 0.835712s) 2024-04-22T21:33:54.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:33:54.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T21:33:54.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: Manager daemon y is now available 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:33:54.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:54 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:33:54.836 INFO:teuthology.orchestra.run.smithi165.stderr:mgr epoch 5 is available 2024-04-22T21:33:54.836 INFO:teuthology.orchestra.run.smithi165.stderr:Setting orchestrator backend to cephadm... 2024-04-22T21:33:55.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:55 smithi165 ceph-mon[21833]: mgrmap e7: y(active, since 1.84026s) 2024-04-22T21:33:55.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:55 smithi165 ceph-mon[21833]: [22/Apr/2024:21:33:54] ENGINE Bus STARTING 2024-04-22T21:33:55.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:55 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:33:55.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:55 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:56 smithi165 ceph-mon[21833]: [22/Apr/2024:21:33:54] ENGINE Serving on https://172.21.15.165:7150 2024-04-22T21:33:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:56 smithi165 ceph-mon[21833]: [22/Apr/2024:21:33:54] ENGINE Bus STARTED 2024-04-22T21:33:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:56 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:33:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:56 smithi165 ceph-mon[21833]: mgrmap e8: y(active, since 2s) 2024-04-22T21:33:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:56 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:33:56.596 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: value unchanged 2024-04-22T21:33:57.164 INFO:teuthology.orchestra.run.smithi165.stderr:Generating ssh key... 2024-04-22T21:33:57.481 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:57 smithi165 ceph-mon[21833]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:33:58.433 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-mon[21833]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:33:58.433 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: Generating public/private rsa key pair. 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: Your identification has been saved in /tmp/tmp76o2vble/key. 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: Your public key has been saved in /tmp/tmp76o2vble/key.pub. 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: The key fingerprint is: 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: SHA256:lEBD+qLAd2BTib2ewXaXfiYXWQjLuhoInpZ6q0+PUbw ceph-dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: The key's randomart image is: 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: +---[RSA 3072]----+ 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: | o+* .. . | 2024-04-22T21:33:58.434 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: | ..+ o.... . | 2024-04-22T21:33:58.435 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: | +o . oo. o | 2024-04-22T21:33:58.435 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: |. ..o* o.o o | 2024-04-22T21:33:58.435 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: |.o .*.=.S . | 2024-04-22T21:33:58.435 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: |..=oo= .o + | 2024-04-22T21:33:58.435 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: | =+.E. . = | 2024-04-22T21:33:58.435 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: |o..+ o | 2024-04-22T21:33:58.435 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: |o++... | 2024-04-22T21:33:58.435 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:33:58 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: +----[SHA256]-----+ 2024-04-22T21:33:59.270 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChorf9v3I1TTpnzVA387pa1JJzYxRIOV768DFwAkJ9SxQTm+rLBFqlvbwmGkx7/z6n7FwEcpFHFJ1TDhuxGcORdqlBXYWCBPmH2+dYDNKyTqs1fcCibPqfrwkTOng+qMud9pubVBs/Ws4SvXjd7wmITAK/g997QP2OPKpsGq271r+QbPI7mD+eLXhxrn48x2WqqDNZW0HI3Y8l3wY8g6qdoEXLtEuyh4MX84gakNkVbZLIwzJp6gFWizNh9Y4FXDsAFiE50pmBcNTzO1y9EaK2AneDXKRfog8NNVFF7pHRSHBtQ3n1R/MfcFwNIvot+7Qf1WIAc1Wgmw08BW9aCphxVgbYYcwZJM9Ns0xsZsbn9EmIfNycHdMvt9w9XFoY1JsjACC2cZzSJOIgLuW3ctGIIz4j43nFZwI+xmxvAVitW0Ut8yNItM6+4eULTCCH4MGLDg5hYvMEWr8qYK0hY+JE7eX2nK8M/iAFA5tHfCHF/T8Jjk11M1NQft31bC3cWxM= ceph-dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:33:59.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:59 smithi165 ceph-mon[21833]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:33:59.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:33:59 smithi165 ceph-mon[21833]: Generating ssh key... 2024-04-22T21:33:59.750 INFO:teuthology.orchestra.run.smithi165.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T21:33:59.750 INFO:teuthology.orchestra.run.smithi165.stderr:Adding key to root@localhost authorized_keys... 2024-04-22T21:33:59.750 INFO:teuthology.orchestra.run.smithi165.stderr:Adding host smithi165... 2024-04-22T21:34:00.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:00 smithi165 ceph-mon[21833]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:01.188 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: Added host 'smithi165' with addr '172.21.15.165' 2024-04-22T21:34:01.546 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:01 smithi165 ceph-mon[21833]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "addr": "172.21.15.165", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:01.546 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:01 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:34:01.547 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:01 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:01.798 INFO:teuthology.orchestra.run.smithi165.stderr:Deploying unmanaged mon service... 2024-04-22T21:34:02.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:02 smithi165 ceph-mon[21833]: Deploying cephadm binary to smithi165 2024-04-22T21:34:02.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:02 smithi165 ceph-mon[21833]: Added host smithi165 2024-04-22T21:34:02.493 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-22T21:34:03.716 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:03 smithi165 ceph-mon[21833]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:03.716 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:03 smithi165 ceph-mon[21833]: Saving service mon spec with placement count:5 2024-04-22T21:34:03.716 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:03 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:34:04.096 INFO:teuthology.orchestra.run.smithi165.stderr:Deploying unmanaged mgr service... 2024-04-22T21:34:04.793 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-22T21:34:05.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:04 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:34:05.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:04 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:34:05.955 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:05 smithi165 ceph-mon[21833]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:05.955 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:05 smithi165 ceph-mon[21833]: Saving service mgr spec with placement count:2 2024-04-22T21:34:05.955 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:05 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:34:05.955 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:05 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:34:07.777 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:07 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/610409764' entity='client.admin' 2024-04-22T21:34:07.777 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:07 smithi165 ceph-mon[21833]: from='mgr.14118 172.21.15.165:0/438147198' entity='mgr.y' 2024-04-22T21:34:08.431 INFO:teuthology.orchestra.run.smithi165.stderr:Enabling the dashboard module... 2024-04-22T21:34:09.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:08 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3916912408' entity='client.admin' 2024-04-22T21:34:10.017 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:10 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3363676955' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T21:34:10.289 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:10 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ignoring --setuser ceph since I am not root 2024-04-22T21:34:10.289 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:10 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ignoring --setgroup ceph since I am not root 2024-04-22T21:34:10.290 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:10 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:10.180+0000 7f14a7393000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T21:34:10.558 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:10 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:10.309+0000 7f14a7393000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:34:10.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:10 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:10.408+0000 7f14a7393000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T21:34:10.861 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:10 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:10.721+0000 7f14a7393000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T21:34:10.862 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:10 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:10.804+0000 7f14a7393000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T21:34:11.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:11 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3363676955' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T21:34:11.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:11 smithi165 ceph-mon[21833]: mgrmap e9: y(active, since 17s) 2024-04-22T21:34:11.368 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: { 2024-04-22T21:34:11.368 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-22T21:34:11.368 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "available": true, 2024-04-22T21:34:11.368 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-22T21:34:11.368 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-22T21:34:11.368 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: } 2024-04-22T21:34:11.958 INFO:teuthology.orchestra.run.smithi165.stderr:Waiting for the mgr to restart... 2024-04-22T21:34:11.958 INFO:teuthology.orchestra.run.smithi165.stderr:Waiting for mgr epoch 9... 2024-04-22T21:34:12.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:12 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2662634328' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T21:34:12.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:11 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:11.943+0000 7f14a7393000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T21:34:12.367 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:12 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:12.115+0000 7f14a7393000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T21:34:12.368 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:12 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:12.200+0000 7f14a7393000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T21:34:12.690 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:12 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:12.524+0000 7f14a7393000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T21:34:12.690 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:12 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:12.614+0000 7f14a7393000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T21:34:12.690 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:12 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:12.707+0000 7f14a7393000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T21:34:13.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:12 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:12.947+0000 7f14a7393000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:34:13.808 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:13 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:13.383+0000 7f14a7393000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T21:34:14.240 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:13 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:13.869+0000 7f14a7393000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T21:34:14.240 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:13 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:13.963+0000 7f14a7393000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T21:34:14.240 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:14 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:14.257+0000 7f14a7393000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T21:34:14.558 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:14 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:14.347+0000 7f14a7393000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T21:34:15.058 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:14 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:14.760+0000 7f14a7393000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T21:34:15.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:14 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:14.890+0000 7f14a7393000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T21:34:15.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:14 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:14.982+0000 7f14a7393000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T21:34:15.558 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:15 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:15.096+0000 7f14a7393000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T21:34:15.559 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:15 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:15.184+0000 7f14a7393000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T21:34:16.058 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:15 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:15.733+0000 7f14a7393000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T21:34:16.495 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:16 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:34:16.113+0000 7f14a7393000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T21:34:16.496 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:16 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:34:16] ENGINE Bus STARTING 2024-04-22T21:34:16.808 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:16 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:34:16] ENGINE Serving on https://172.21.15.165:7150 2024-04-22T21:34:16.809 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:34:16 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:34:16] ENGINE Bus STARTED 2024-04-22T21:34:17.164 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: { 2024-04-22T21:34:17.165 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-22T21:34:17.165 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: "initialized": true 2024-04-22T21:34:17.165 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: } 2024-04-22T21:34:17.533 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: Active manager daemon y restarted 2024-04-22T21:34:17.533 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: Activating manager daemon y 2024-04-22T21:34:17.533 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T21:34:17.533 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: mgrmap e10: y(active, starting, since 0.0553391s) 2024-04-22T21:34:17.533 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:34:17.534 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T21:34:17.534 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:34:17.534 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:34:17.534 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:34:17.534 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: Manager daemon y is now available 2024-04-22T21:34:17.534 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:17.534 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:17.534 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:34:17.535 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:34:17.535 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: [22/Apr/2024:21:34:16] ENGINE Bus STARTING 2024-04-22T21:34:17.535 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: [22/Apr/2024:21:34:16] ENGINE Serving on https://172.21.15.165:7150 2024-04-22T21:34:17.535 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: [22/Apr/2024:21:34:16] ENGINE Bus STARTED 2024-04-22T21:34:17.535 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:17.836 INFO:teuthology.orchestra.run.smithi165.stderr:mgr epoch 9 is available 2024-04-22T21:34:17.836 INFO:teuthology.orchestra.run.smithi165.stderr:Generating a dashboard self-signed certificate... 2024-04-22T21:34:18.405 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:18 smithi165 ceph-mon[21833]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T21:34:18.405 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:18 smithi165 ceph-mon[21833]: mgrmap e11: y(active, since 1.06031s) 2024-04-22T21:34:18.405 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:18 smithi165 ceph-mon[21833]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T21:34:18.405 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:18 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:18.593 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-22T21:34:19.201 INFO:teuthology.orchestra.run.smithi165.stderr:Creating initial admin user... 2024-04-22T21:34:19.652 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:19.652 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:19 smithi165 ceph-mon[21833]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:19.652 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:19.653 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:19.653 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:19 smithi165 ceph-mon[21833]: mgrmap e12: y(active, since 2s) 2024-04-22T21:34:20.102 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$FVl1x7WSW.E1egNIW5tzI..cqRg5qaekGxNPxPeS26ItokGuIh3ga", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713821660, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T21:34:20.445 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:20 smithi165 ceph-mon[21833]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:20.445 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:20 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:21.465 INFO:teuthology.orchestra.run.smithi165.stderr:Fetching dashboard port number... 2024-04-22T21:34:22.037 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: 8443 2024-04-22T21:34:22.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:22 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3042145266' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T21:34:22.733 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: disabled 2024-04-22T21:34:22.740 INFO:teuthology.orchestra.run.smithi165.stderr:systemctl: inactive 2024-04-22T21:34:22.740 INFO:teuthology.orchestra.run.smithi165.stderr:firewalld.service is not enabled 2024-04-22T21:34:22.740 INFO:teuthology.orchestra.run.smithi165.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T21:34:22.742 INFO:teuthology.orchestra.run.smithi165.stderr:Ceph Dashboard is now available at: 2024-04-22T21:34:22.742 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:34:22.743 INFO:teuthology.orchestra.run.smithi165.stderr: URL: https://smithi165:8443/ 2024-04-22T21:34:22.743 INFO:teuthology.orchestra.run.smithi165.stderr: User: admin 2024-04-22T21:34:22.743 INFO:teuthology.orchestra.run.smithi165.stderr: Password: qp7qi5r2hu 2024-04-22T21:34:22.743 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:34:22.743 INFO:teuthology.orchestra.run.smithi165.stderr:Enabling autotune for osd_memory_target 2024-04-22T21:34:24.240 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:24 smithi165 ceph-mon[21833]: mgrmap e13: y(active, since 6s) 2024-04-22T21:34:24.724 INFO:teuthology.orchestra.run.smithi165.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-22T21:34:25.506 INFO:teuthology.orchestra.run.smithi165.stderr:You can access the Ceph CLI with: 2024-04-22T21:34:25.506 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr: ceph telemetry on 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr:For more information see: 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:34:25.507 INFO:teuthology.orchestra.run.smithi165.stderr:Bootstrap complete. 2024-04-22T21:34:25.547 INFO:tasks.cephadm:Fetching config... 2024-04-22T21:34:25.548 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:34:25.548 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T21:34:25.564 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T21:34:25.564 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:34:25.564 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T21:34:25.620 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T21:34:25.620 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:34:25.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-22T21:34:25.685 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T21:34:25.685 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:34:25.685 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T21:34:25.741 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T21:34:25.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChorf9v3I1TTpnzVA387pa1JJzYxRIOV768DFwAkJ9SxQTm+rLBFqlvbwmGkx7/z6n7FwEcpFHFJ1TDhuxGcORdqlBXYWCBPmH2+dYDNKyTqs1fcCibPqfrwkTOng+qMud9pubVBs/Ws4SvXjd7wmITAK/g997QP2OPKpsGq271r+QbPI7mD+eLXhxrn48x2WqqDNZW0HI3Y8l3wY8g6qdoEXLtEuyh4MX84gakNkVbZLIwzJp6gFWizNh9Y4FXDsAFiE50pmBcNTzO1y9EaK2AneDXKRfog8NNVFF7pHRSHBtQ3n1R/MfcFwNIvot+7Qf1WIAc1Wgmw08BW9aCphxVgbYYcwZJM9Ns0xsZsbn9EmIfNycHdMvt9w9XFoY1JsjACC2cZzSJOIgLuW3ctGIIz4j43nFZwI+xmxvAVitW0Ut8yNItM6+4eULTCCH4MGLDg5hYvMEWr8qYK0hY+JE7eX2nK8M/iAFA5tHfCHF/T8Jjk11M1NQft31bC3cWxM= ceph-dccb09a4-00ef-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T21:34:25.801 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:25 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2884587597' entity='client.admin' 2024-04-22T21:34:25.831 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChorf9v3I1TTpnzVA387pa1JJzYxRIOV768DFwAkJ9SxQTm+rLBFqlvbwmGkx7/z6n7FwEcpFHFJ1TDhuxGcORdqlBXYWCBPmH2+dYDNKyTqs1fcCibPqfrwkTOng+qMud9pubVBs/Ws4SvXjd7wmITAK/g997QP2OPKpsGq271r+QbPI7mD+eLXhxrn48x2WqqDNZW0HI3Y8l3wY8g6qdoEXLtEuyh4MX84gakNkVbZLIwzJp6gFWizNh9Y4FXDsAFiE50pmBcNTzO1y9EaK2AneDXKRfog8NNVFF7pHRSHBtQ3n1R/MfcFwNIvot+7Qf1WIAc1Wgmw08BW9aCphxVgbYYcwZJM9Ns0xsZsbn9EmIfNycHdMvt9w9XFoY1JsjACC2cZzSJOIgLuW3ctGIIz4j43nFZwI+xmxvAVitW0Ut8yNItM6+4eULTCCH4MGLDg5hYvMEWr8qYK0hY+JE7eX2nK8M/iAFA5tHfCHF/T8Jjk11M1NQft31bC3cWxM= ceph-dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:34:25.843 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChorf9v3I1TTpnzVA387pa1JJzYxRIOV768DFwAkJ9SxQTm+rLBFqlvbwmGkx7/z6n7FwEcpFHFJ1TDhuxGcORdqlBXYWCBPmH2+dYDNKyTqs1fcCibPqfrwkTOng+qMud9pubVBs/Ws4SvXjd7wmITAK/g997QP2OPKpsGq271r+QbPI7mD+eLXhxrn48x2WqqDNZW0HI3Y8l3wY8g6qdoEXLtEuyh4MX84gakNkVbZLIwzJp6gFWizNh9Y4FXDsAFiE50pmBcNTzO1y9EaK2AneDXKRfog8NNVFF7pHRSHBtQ3n1R/MfcFwNIvot+7Qf1WIAc1Wgmw08BW9aCphxVgbYYcwZJM9Ns0xsZsbn9EmIfNycHdMvt9w9XFoY1JsjACC2cZzSJOIgLuW3ctGIIz4j43nFZwI+xmxvAVitW0Ut8yNItM6+4eULTCCH4MGLDg5hYvMEWr8qYK0hY+JE7eX2nK8M/iAFA5tHfCHF/T8Jjk11M1NQft31bC3cWxM= ceph-dccb09a4-00ef-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T21:34:25.883 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChorf9v3I1TTpnzVA387pa1JJzYxRIOV768DFwAkJ9SxQTm+rLBFqlvbwmGkx7/z6n7FwEcpFHFJ1TDhuxGcORdqlBXYWCBPmH2+dYDNKyTqs1fcCibPqfrwkTOng+qMud9pubVBs/Ws4SvXjd7wmITAK/g997QP2OPKpsGq271r+QbPI7mD+eLXhxrn48x2WqqDNZW0HI3Y8l3wY8g6qdoEXLtEuyh4MX84gakNkVbZLIwzJp6gFWizNh9Y4FXDsAFiE50pmBcNTzO1y9EaK2AneDXKRfog8NNVFF7pHRSHBtQ3n1R/MfcFwNIvot+7Qf1WIAc1Wgmw08BW9aCphxVgbYYcwZJM9Ns0xsZsbn9EmIfNycHdMvt9w9XFoY1JsjACC2cZzSJOIgLuW3ctGIIz4j43nFZwI+xmxvAVitW0Ut8yNItM6+4eULTCCH4MGLDg5hYvMEWr8qYK0hY+JE7eX2nK8M/iAFA5tHfCHF/T8Jjk11M1NQft31bC3cWxM= ceph-dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:34:25.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T21:34:28.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:28.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:28.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:28.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:28.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:30.057 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T21:34:30.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T21:34:30.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:29 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/860102529' entity='client.admin' 2024-04-22T21:34:30.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:29 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:30.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:29 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:32.550 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2024-04-22T21:34:32.551 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-22T21:34:32.551 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2024-04-22T21:34:32.569 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-22T21:34:32.569 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:34:32.627 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2024-04-22T21:34:32.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch host add smithi188 2024-04-22T21:34:33.148 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:32 smithi165 ceph-mon[21833]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:33.148 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:33.148 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:33.148 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:33.148 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:33.148 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:32 smithi165 ceph-mon[21833]: Updating smithi165:/etc/ceph/ceph.conf 2024-04-22T21:34:33.148 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:33.148 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:34.121 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:33 smithi165 ceph-mon[21833]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:34:35.244 INFO:teuthology.orchestra.run.smithi165.stdout:Added host 'smithi188' with addr '172.21.15.188' 2024-04-22T21:34:35.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:35 smithi165 ceph-mon[21833]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:35.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:35 smithi165 ceph-mon[21833]: Deploying cephadm binary to smithi188 2024-04-22T21:34:35.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T21:34:36.447 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:36 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:36.447 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:36 smithi165 ceph-mon[21833]: Added host smithi188 2024-04-22T21:34:36.447 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:36 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:36.447 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:36 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:36.447 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:36 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:36.447 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:36 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:36.447 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:36 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:37.722 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:34:37.722 INFO:teuthology.orchestra.run.smithi165.stdout:[{"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2024-04-22T21:34:38.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:38 smithi165 ceph-mon[21833]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:38.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:38.331 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T21:34:38.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T21:34:39.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:39 smithi165 ceph-mon[21833]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:34:40.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:40 smithi165 ceph-mon[21833]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:40.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:40.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:40.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:34:40.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:41.046 INFO:teuthology.orchestra.run.smithi165.stderr:adjusted tunables profile to default 2024-04-22T21:34:41.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:41 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/470270075' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T21:34:41.636 INFO:tasks.cephadm:Adding mon.a on smithi165 2024-04-22T21:34:41.636 INFO:tasks.cephadm:Adding mon.c on smithi165 2024-04-22T21:34:41.636 INFO:tasks.cephadm:Adding mon.b on smithi188 2024-04-22T21:34:41.636 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi165:172.21.15.165=a;smithi165:[v2:172.21.15.165:3301,v1:172.21.15.165:6790]=c;smithi188:172.21.15.188=b' 2024-04-22T21:34:42.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:42 smithi165 ceph-mon[21833]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:42.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:42 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/470270075' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T21:34:42.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:42 smithi165 ceph-mon[21833]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:34:43.544 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2024-04-22T21:34:44.229 DEBUG:teuthology.orchestra.run.smithi165:mon.c> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.c.service 2024-04-22T21:34:44.231 DEBUG:teuthology.orchestra.run.smithi188:mon.b> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.b.service 2024-04-22T21:34:44.233 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T21:34:44.234 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:34:44.249 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:44 smithi165 ceph-mon[21833]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:44.249 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:44 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:44.249 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:44 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:44.249 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:44 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:44.249 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:44 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:44.249 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:44 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:44.250 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:44 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:44.250 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:44 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:34:44.250 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:44 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:45.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:45 smithi165 ceph-mon[21833]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi165:172.21.15.165=a;smithi165:[v2:172.21.15.165:3301,v1:172.21.15.165:6790]=c;smithi188:172.21.15.188=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:45.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:45 smithi165 ceph-mon[21833]: Saving service mon spec with placement smithi165:172.21.15.165=a;smithi165:[v2:172.21.15.165:3301,v1:172.21.15.165:6790]=c;smithi188:172.21.15.188=b;count:3 2024-04-22T21:34:45.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:45 smithi165 ceph-mon[21833]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-22T21:34:45.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:45 smithi165 ceph-mon[21833]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:34:45.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:45 smithi165 ceph-mon[21833]: Deploying daemon mon.c on smithi165 2024-04-22T21:34:46.115 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-22T21:34:46.115 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"dccb09a4-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:33:19.874210Z","created":"2024-04-22T21:33:19.874210Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T21:34:46.117 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-22T21:34:46.296 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:46 smithi165 ceph-mon[21833]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:46.985 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:46 smithi165 podman[26531]: 2024-04-22 21:34:46.89337747 +0000 UTC m=+0.129274854 container create 8798a7ddca5671eda62940694f9b3d5ac98c3e4414a72983ab5d747d5cd12746 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-c, io.openshift.tags=base centos centos-stream, architecture=x86_64, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vendor=Red Hat, Inc., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-22T21:34:47.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/1275250360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:34:47.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 podman[26531]: 2024-04-22 21:34:47.079528598 +0000 UTC m=+0.315425982 container init 8798a7ddca5671eda62940694f9b3d5ac98c3e4414a72983ab5d747d5cd12746 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.openshift.tags=base centos centos-stream, version=8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, release=754, io.openshift.expose-services=, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T21:34:47.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 podman[26531]: 2024-04-22 21:34:47.082669813 +0000 UTC m=+0.318567198 container start 8798a7ddca5671eda62940694f9b3d5ac98c3e4414a72983ab5d747d5cd12746 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vcs-type=git, release=754, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, name=centos-stream, ceph=True, vendor=Red Hat, Inc., version=8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-22T21:34:47.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T21:34:47.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-22T21:34:47.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: pidfile_write: ignore empty --pid-file 2024-04-22T21:34:47.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: load: jerasure load: lrc 2024-04-22T21:34:47.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: RocksDB version: 6.15.5 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Compile date Apr 18 2022 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: DB SUMMARY 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: DB Session ID: Q4G8G5SGXGTGZWXCQJN7 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: CURRENT file: CURRENT 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.error_if_exists: 0 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.create_if_missing: 0 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.paranoid_checks: 1 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.env: 0x55c8e82c6860 2024-04-22T21:34:47.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.fs: Posix File System 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.info_log: 0x55c8ea8e7dc0 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.statistics: (nil) 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.use_fsync: 0 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_log_file_size: 0 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.allow_fallocate: 1 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.use_direct_reads: 0 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T21:34:47.311 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T21:34:47.312 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.db_log_dir: 2024-04-22T21:34:47.312 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-04-22T21:34:47.312 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T21:34:47.312 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T21:34:47.312 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T21:34:47.312 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T21:34:47.312 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T21:34:47.312 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.write_buffer_manager: 0x55c8ea9d8270 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T21:34:47.314 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.unordered_write: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.row_cache: None 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.wal_filter: None 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.preserve_deletes: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.two_write_queues: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.atomic_flush: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T21:34:47.315 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.log_readahead_size: 0 2024-04-22T21:34:47.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T21:34:47.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T21:34:47.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T21:34:47.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T21:34:47.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T21:34:47.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T21:34:47.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_background_jobs: 2 2024-04-22T21:34:47.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_background_compactions: -1 2024-04-22T21:34:47.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_subcompactions: 1 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_open_files: -1 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_background_flushes: -1 2024-04-22T21:34:47.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Compression algorithms supported: 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: kZSTD supported: 0 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: kXpressCompression supported: 0 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: kLZ4Compression supported: 1 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: kBZip2Compression supported: 0 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: kZlibCompression supported: 1 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: kSnappyCompression supported: 1 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.merge_operator: 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_filter: None 2024-04-22T21:34:47.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c8ea8b3d00) 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: cache_index_and_filter_blocks: 1 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: pin_top_level_index_and_filter: 1 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: index_type: 0 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: data_block_index_type: 0 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: index_shortening: 1 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: hash_index_allow_collision: 1 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: checksum: 1 2024-04-22T21:34:47.319 INFO:journalctl@ceph.mon.c.smithi165.stdout: no_block_cache: 0 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_cache: 0x55c8ea91e170 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_cache_name: BinnedLRUCache 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_cache_options: 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: capacity : 536870912 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: num_shard_bits : 4 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: strict_capacity_limit : 0 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: high_pri_pool_ratio: 0.000 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_cache_compressed: (nil) 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: persistent_cache: (nil) 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_size: 4096 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_size_deviation: 10 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_restart_interval: 16 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: index_block_restart_interval: 1 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: metadata_block_size: 4096 2024-04-22T21:34:47.320 INFO:journalctl@ceph.mon.c.smithi165.stdout: partition_filters: 0 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout: use_delta_encoding: 1 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout: whole_key_filtering: 1 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout: verify_compression: 0 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout: read_amp_bytes_per_bit: 0 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout: format_version: 4 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout: enable_index_compression: 1 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_align: 0 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compression: NoCompression 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T21:34:47.321 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.num_levels: 7 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T21:34:47.322 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T21:34:47.323 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T21:34:47.324 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.table_properties_collectors: 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.inplace_update_support: 0 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T21:34:47.325 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T21:34:47.326 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T21:34:47.326 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.bloom_locality: 0 2024-04-22T21:34:47.326 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.max_successive_merges: 0 2024-04-22T21:34:47.326 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T21:34:47.326 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T21:34:47.326 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T21:34:47.326 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T21:34:47.326 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.ttl: 2592000 2024-04-22T21:34:47.326 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.enable_blob_files: false 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.min_blob_size: 0 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821687108453, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821687108751, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713821687, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c05dc2c5-53bc-4825-bb07-89b10d9f1d69", "db_session_id": "Q4G8G5SGXGTGZWXCQJN7"}} 2024-04-22T21:34:47.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821687109052, "job": 1, "event": "recovery_finished"} 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55c8ea904a80 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: DB pointer 0x55c8ea914000 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** DB Stats ** 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T21:34:47.328 INFO:journalctl@ceph.mon.c.smithi165.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** Compaction Stats [default] ** 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** Compaction Stats [default] ** 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:34:47.329 INFO:journalctl@ceph.mon.c.smithi165.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** Compaction Stats [default] ** 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:34:47.330 INFO:journalctl@ceph.mon.c.smithi165.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** Compaction Stats [default] ** 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:34:47.331 INFO:journalctl@ceph.mon.c.smithi165.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: using public_addrv [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: starting mon.c rank -1 at public addrs [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] at bind addrs [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(???) e0 preinit fsid dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).mds e1 new map 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout: e1 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout: legacy client fscid: -1 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout: No filesystems configured 2024-04-22T21:34:47.332 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/470270075' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:47.333 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi165:172.21.15.165=a;smithi165:[v2:172.21.15.165:3301,v1:172.21.15.165:6790]=c;smithi188:172.21.15.188=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: Saving service mon spec with placement smithi165:172.21.15.165=a;smithi165:[v2:172.21.15.165:3301,v1:172.21.15.165:6790]=c;smithi188:172.21.15.188=b;count:3 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: Deploying daemon mon.c on smithi165 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/1275250360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 ceph-mon[26549]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-22T21:34:47.334 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 bash[26531]: 8798a7ddca5671eda62940694f9b3d5ac98c3e4414a72983ab5d747d5cd12746 2024-04-22T21:34:47.335 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:47 smithi165 systemd[1]: Started Ceph mon.c for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:34:47.679 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T21:34:47.679 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:34:52.504 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-22T21:34:52.505 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"dccb09a4-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:34:47.129961Z","created":"2024-04-22T21:33:19.874210Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3301","nonce":0},{"type":"v1","addr":"172.21.15.165:6790","nonce":0}]},"addr":"172.21.15.165:6790/0","public_addr":"172.21.15.165:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T21:34:52.506 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: mon.a calling monitor election 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: mon.c calling monitor election 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: monmap e2: 2 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0]} 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: fsmap 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: mgrmap e13: y(active, since 36s) 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: overall HEALTH_OK 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: mon.a calling monitor election 2024-04-22T21:34:52.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: mon.c calling monitor election 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: monmap e2: 2 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0]} 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: fsmap 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: mgrmap e13: y(active, since 36s) 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: overall HEALTH_OK 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:52.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:34:52.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:53 smithi165 ceph-mon[21833]: Deploying daemon mon.b on smithi188 2024-04-22T21:34:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:53 smithi165 ceph-mon[21833]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:53 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/2636951157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:34:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:53 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:53.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:53 smithi165 ceph-mon[26549]: Deploying daemon mon.b on smithi188 2024-04-22T21:34:53.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:53 smithi165 ceph-mon[26549]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:53.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:53 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/2636951157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:34:53.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:53 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:54.995 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T21:34:54.995 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T21:34:55.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:34:55 smithi165 ceph-mon[21833]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:55.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:34:55 smithi165 ceph-mon[26549]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:56.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:55 smithi188 systemd[1]: Starting Ceph mon.b for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:34:56.670 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:56 smithi188 podman[22643]: 2024-04-22 21:34:56.453615047 +0000 UTC m=+0.023355629 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:34:56.964 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:56 smithi188 podman[22643]: 2024-04-22 21:34:56.796435307 +0000 UTC m=+0.366175848 container create 1aa8711a0f5e7dec13b36be36f3270e20bb63c8489927488977193a98074f41d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-b, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vcs-type=git, io.buildah.version=1.19.8, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, release=754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T21:34:57.219 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 podman[22643]: 2024-04-22 21:34:57.158064438 +0000 UTC m=+0.727804982 container init 1aa8711a0f5e7dec13b36be36f3270e20bb63c8489927488977193a98074f41d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-b, RELEASE=HEAD, GIT_BRANCH=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.component=centos-stream-container, ceph=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.19.8, version=8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, release=754) 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 podman[22643]: 2024-04-22 21:34:57.161356307 +0000 UTC m=+0.731096858 container start 1aa8711a0f5e7dec13b36be36f3270e20bb63c8489927488977193a98074f41d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-b, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, version=8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, name=centos-stream, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, architecture=x86_64, maintainer=Guillaume Abrioux ) 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: pidfile_write: ignore empty --pid-file 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: load: jerasure load: lrc 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: RocksDB version: 6.15.5 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Compile date Apr 18 2022 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: DB SUMMARY 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: DB Session ID: 1BRCT0JIU1ZBPFLK9A9E 2024-04-22T21:34:57.220 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: CURRENT file: CURRENT 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.error_if_exists: 0 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.create_if_missing: 0 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.paranoid_checks: 1 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.env: 0x5607004c6860 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.fs: Posix File System 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.info_log: 0x560701bc1dc0 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.statistics: (nil) 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.use_fsync: 0 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_log_file_size: 0 2024-04-22T21:34:57.221 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.allow_fallocate: 1 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.use_direct_reads: 0 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.db_log_dir: 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T21:34:57.222 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T21:34:57.223 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T21:34:57.223 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T21:34:57.224 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T21:34:57.224 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T21:34:57.224 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.write_buffer_manager: 0x560701cb2240 2024-04-22T21:34:57.224 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T21:34:57.224 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T21:34:57.224 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T21:34:57.224 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.unordered_write: 0 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.row_cache: None 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.wal_filter: None 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T21:34:57.225 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.preserve_deletes: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.two_write_queues: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.atomic_flush: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.log_readahead_size: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T21:34:57.226 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_background_jobs: 2 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_background_compactions: -1 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_subcompactions: 1 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_open_files: -1 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_background_flushes: -1 2024-04-22T21:34:57.227 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Compression algorithms supported: 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: kZSTD supported: 0 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: kXpressCompression supported: 0 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: kLZ4Compression supported: 1 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: kBZip2Compression supported: 0 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: kZlibCompression supported: 1 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: kSnappyCompression supported: 1 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.merge_operator: 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_filter: None 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T21:34:57.228 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560701b8dd00) 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: cache_index_and_filter_blocks: 1 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: pin_top_level_index_and_filter: 1 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: index_type: 0 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: data_block_index_type: 0 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: index_shortening: 1 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: hash_index_allow_collision: 1 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: checksum: 1 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: no_block_cache: 0 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: block_cache: 0x560701bf8170 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: block_cache_name: BinnedLRUCache 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: block_cache_options: 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: capacity : 536870912 2024-04-22T21:34:57.229 INFO:journalctl@ceph.mon.b.smithi188.stdout: num_shard_bits : 4 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: strict_capacity_limit : 0 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: high_pri_pool_ratio: 0.000 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: block_cache_compressed: (nil) 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: persistent_cache: (nil) 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: block_size: 4096 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: block_size_deviation: 10 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: block_restart_interval: 16 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: index_block_restart_interval: 1 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: metadata_block_size: 4096 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: partition_filters: 0 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: use_delta_encoding: 1 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: whole_key_filtering: 1 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: verify_compression: 0 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: read_amp_bytes_per_bit: 0 2024-04-22T21:34:57.230 INFO:journalctl@ceph.mon.b.smithi188.stdout: format_version: 4 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout: enable_index_compression: 1 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout: block_align: 0 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compression: NoCompression 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.num_levels: 7 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T21:34:57.231 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T21:34:57.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T21:34:57.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.table_properties_collectors: 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.inplace_update_support: 0 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T21:34:57.234 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.bloom_locality: 0 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.max_successive_merges: 0 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.ttl: 2592000 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.enable_blob_files: false 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.min_blob_size: 0 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T21:34:57.235 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821697188012, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821697188322, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713821697, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "321199a5-564f-4d75-b6be-e3cfcb07092b", "db_session_id": "1BRCT0JIU1ZBPFLK9A9E"}} 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713821697188653, "job": 1, "event": "recovery_finished"} 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x560701bdea80 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: DB pointer 0x560701bee000 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T21:34:57.236 INFO:journalctl@ceph.mon.b.smithi188.stdout: ** DB Stats ** 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-22T21:34:57.237 INFO:journalctl@ceph.mon.b.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: 2024-04-22T21:34:57.238 INFO:journalctl@ceph.mon.b.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T21:34:57.239 INFO:journalctl@ceph.mon.b.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: using public_addr v2:172.21.15.188:0/0 -> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: starting mon.b rank -1 at public addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] at bind addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(???) e0 preinit fsid dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).mds e1 new map 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: e1 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: legacy client fscid: -1 2024-04-22T21:34:57.240 INFO:journalctl@ceph.mon.b.smithi188.stdout: 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout: No filesystems configured 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/470270075' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:57.241 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi165:172.21.15.165=a;smithi165:[v2:172.21.15.165:3301,v1:172.21.15.165:6790]=c;smithi188:172.21.15.188=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: Saving service mon spec with placement smithi165:172.21.15.165=a;smithi165:[v2:172.21.15.165:3301,v1:172.21.15.165:6790]=c;smithi188:172.21.15.188=b;count:3 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: Deploying daemon mon.c on smithi165 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/1275250360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:34:57.242 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.a calling monitor election 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.c calling monitor election 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: monmap e2: 2 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],c=[v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0]} 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: fsmap 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mgrmap e13: y(active, since 36s) 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: overall HEALTH_OK 2024-04-22T21:34:57.243 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: Deploying daemon mon.b on smithi188 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/2636951157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 ceph-mon[22698]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 bash[22643]: 1aa8711a0f5e7dec13b36be36f3270e20bb63c8489927488977193a98074f41d 2024-04-22T21:34:57.244 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:34:57 smithi188 systemd[1]: Started Ceph mon.b for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:35:02.200 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-22T21:35:02.200 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":3,"fsid":"dccb09a4-00ef-11ef-bc93-c7b262605968","modified":"2024-04-22T21:34:57.207628Z","created":"2024-04-22T21:33:19.874210Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3301","nonce":0},{"type":"v1","addr":"172.21.15.165:6790","nonce":0}]},"addr":"172.21.15.165:6790/0","public_addr":"172.21.15.165:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T21:35:02.203 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 3 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: mon.a calling monitor election 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: mon.c calling monitor election 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/2894297030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: monmap e3: 3 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],b=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0],c=[v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0]} 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: fsmap 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: mgrmap e13: y(active, since 46s) 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: mon.b (rank 2) addr [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] is down (out of quorum) 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:02.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: mon.a calling monitor election 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: mon.c calling monitor election 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/2894297030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: monmap e3: 3 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],b=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0],c=[v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0]} 2024-04-22T21:35:02.561 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: fsmap 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: mgrmap e13: y(active, since 46s) 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: mon.b (rank 2) addr [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] is down (out of quorum) 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:02.562 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:02.781 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T21:35:02.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T21:35:03.479 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:03 smithi165 ceph-mon[26549]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:03.479 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:03 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:03.479 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:03 smithi165 ceph-mon[21833]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:03.479 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:03 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:04.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mon.a calling monitor election 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mon.c calling monitor election 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/2894297030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: monmap e3: 3 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],b=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0],c=[v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0]} 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: fsmap 2024-04-22T21:35:04.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mgrmap e13: y(active, since 46s) 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mon.b (rank 2) addr [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] is down (out of quorum) 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:04.510 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:05.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mon.b calling monitor election 2024-04-22T21:35:05.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mon.b calling monitor election 2024-04-22T21:35:05.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mon.c calling monitor election 2024-04-22T21:35:05.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mon.a calling monitor election 2024-04-22T21:35:05.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T21:35:05.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: monmap e3: 3 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],b=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0],c=[v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0]} 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: fsmap 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: mgrmap e13: y(active, since 48s) 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: Cluster is now healthy 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:04 smithi188 ceph-mon[22698]: overall HEALTH_OK 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: mon.b calling monitor election 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: mon.b calling monitor election 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: mon.c calling monitor election 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: mon.a calling monitor election 2024-04-22T21:35:05.009 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: monmap e3: 3 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],b=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0],c=[v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0]} 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: fsmap 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: mgrmap e13: y(active, since 48s) 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: Cluster is now healthy 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[26549]: overall HEALTH_OK 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: mon.b calling monitor election 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: mon.b calling monitor election 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: mon.c calling monitor election 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: mon.a calling monitor election 2024-04-22T21:35:05.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T21:35:05.011 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: monmap e3: 3 mons at {a=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0],b=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0],c=[v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0]} 2024-04-22T21:35:05.011 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: fsmap 2024-04-22T21:35:05.011 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T21:35:05.011 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: mgrmap e13: y(active, since 48s) 2024-04-22T21:35:05.011 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-22T21:35:05.011 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: Cluster is now healthy 2024-04-22T21:35:05.011 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:04 smithi165 ceph-mon[21833]: overall HEALTH_OK 2024-04-22T21:35:05.917 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[26549]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:05.917 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:05.917 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:05.917 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:35:05.917 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:35:05.917 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:05.918 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[21833]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:05.918 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:05.918 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:05.918 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:35:05.918 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:35:05.918 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:05 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:05.988 INFO:teuthology.orchestra.run.smithi165.stdout:# minimal ceph.conf for dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:35:05.989 INFO:teuthology.orchestra.run.smithi165.stdout:[global] 2024-04-22T21:35:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: fsid = dccb09a4-00ef-11ef-bc93-c7b262605968 2024-04-22T21:35:05.989 INFO:teuthology.orchestra.run.smithi165.stdout: mon_host = [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] 2024-04-22T21:35:06.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:05 smithi188 ceph-mon[22698]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:06.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:05 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:35:06.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:05 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:06.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:05 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:35:06.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:05 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:35:06.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:05 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:07.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:06 smithi188 ceph-mon[22698]: Reconfiguring mon.c (monmap changed)... 2024-04-22T21:35:07.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:06 smithi188 ceph-mon[22698]: Reconfiguring daemon mon.c on smithi165 2024-04-22T21:35:07.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:06 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/2302450389' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:07.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:06 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:07.016 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:06 smithi165 ceph-mon[21833]: Reconfiguring mon.c (monmap changed)... 2024-04-22T21:35:07.016 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:06 smithi165 ceph-mon[21833]: Reconfiguring daemon mon.c on smithi165 2024-04-22T21:35:07.016 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:06 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2302450389' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:07.016 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:06 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:07.016 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:06 smithi165 ceph-mon[26549]: Reconfiguring mon.c (monmap changed)... 2024-04-22T21:35:07.016 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:06 smithi165 ceph-mon[26549]: Reconfiguring daemon mon.c on smithi165 2024-04-22T21:35:07.016 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:06 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/2302450389' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:07.017 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:06 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:07.045 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T21:35:07.045 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:35:07.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T21:35:07.071 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:35:07.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:35:07.135 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-22T21:35:07.135 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T21:35:07.164 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-22T21:35:07.164 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:35:07.232 INFO:tasks.cephadm:Adding mgr.y on smithi165 2024-04-22T21:35:07.232 INFO:tasks.cephadm:Adding mgr.x on smithi188 2024-04-22T21:35:07.232 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi165=y;smithi188=x' 2024-04-22T21:35:07.890 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:07 smithi188 ceph-mon[22698]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:07.890 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:07 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:07.890 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:07 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:35:07.890 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:07 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:35:07.890 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:07 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:08.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[21833]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:08.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:08.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:35:08.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:35:08.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:08.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[26549]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:08.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:08.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T21:35:08.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T21:35:08.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:07 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:08.862 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:08 smithi188 ceph-mon[22698]: Reconfiguring mon.b (monmap changed)... 2024-04-22T21:35:08.862 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:08 smithi188 ceph-mon[22698]: Reconfiguring daemon mon.b on smithi188 2024-04-22T21:35:09.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:08 smithi165 ceph-mon[21833]: Reconfiguring mon.b (monmap changed)... 2024-04-22T21:35:09.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:08 smithi165 ceph-mon[21833]: Reconfiguring daemon mon.b on smithi188 2024-04-22T21:35:09.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:08 smithi165 ceph-mon[26549]: Reconfiguring mon.b (monmap changed)... 2024-04-22T21:35:09.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:08 smithi165 ceph-mon[26549]: Reconfiguring daemon mon.b on smithi188 2024-04-22T21:35:09.158 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mgr update... 2024-04-22T21:35:09.749 DEBUG:teuthology.orchestra.run.smithi188:mgr.x> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mgr.x.service 2024-04-22T21:35:09.751 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T21:35:09.752 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:35:09.752 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T21:35:09.771 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T21:35:09.771 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-04-22T21:35:09.829 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T21:35:09.829 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:35:09.829 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-22T21:35:09.829 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:35:09.829 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:35:09.830 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-22 21:34:46.606943871 +0000 2024-04-22T21:35:09.830 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-22 21:32:28.323045756 +0000 2024-04-22T21:35:09.830 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-22 21:32:28.323045756 +0000 2024-04-22T21:35:09.830 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-22 21:32:28.323045756 +0000 2024-04-22T21:35:09.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T21:35:09.876 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:09 smithi188 ceph-mon[22698]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:09.888 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:09 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:09.897 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-22T21:35:09.897 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-22T21:35:09.897 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000290616 s, 1.8 MB/s 2024-04-22T21:35:09.899 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T21:35:09.957 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-04-22T21:35:10.013 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T21:35:10.013 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:35:10.014 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T21:35:10.014 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:35:10.014 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:35:10.014 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-22 21:34:46.606943871 +0000 2024-04-22T21:35:10.014 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-22 21:32:28.657041213 +0000 2024-04-22T21:35:10.014 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-22 21:32:28.657041213 +0000 2024-04-22T21:35:10.014 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-22 21:32:28.657041213 +0000 2024-04-22T21:35:10.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T21:35:10.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:09 smithi165 ceph-mon[21833]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:10.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:09 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:10.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:09 smithi165 ceph-mon[26549]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:10.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:09 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:10.078 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-22T21:35:10.078 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-22T21:35:10.078 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000214399 s, 2.4 MB/s 2024-04-22T21:35:10.079 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T21:35:10.137 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-04-22T21:35:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T21:35:10.193 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:35:10.193 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-22T21:35:10.194 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:35:10.194 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:35:10.194 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-22 21:34:46.606943871 +0000 2024-04-22T21:35:10.194 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-22 21:32:29.006036466 +0000 2024-04-22T21:35:10.194 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-22 21:32:29.006036466 +0000 2024-04-22T21:35:10.194 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-22 21:32:29.006036466 +0000 2024-04-22T21:35:10.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T21:35:10.258 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-22T21:35:10.258 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-22T21:35:10.258 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000219647 s, 2.3 MB/s 2024-04-22T21:35:10.259 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T21:35:10.315 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-04-22T21:35:10.370 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T21:35:10.370 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:35:10.370 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T21:35:10.370 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:35:10.370 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:35:10.370 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-04-22 21:34:46.606943871 +0000 2024-04-22T21:35:10.370 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-04-22 21:32:29.361031637 +0000 2024-04-22T21:35:10.370 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-04-22 21:32:29.361031637 +0000 2024-04-22T21:35:10.370 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-04-22 21:32:29.361031637 +0000 2024-04-22T21:35:10.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T21:35:10.432 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-04-22T21:35:10.433 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-04-22T21:35:10.433 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000222715 s, 2.3 MB/s 2024-04-22T21:35:10.433 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T21:35:10.489 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-22T21:35:10.489 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T21:35:10.508 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T21:35:10.508 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-04-22T21:35:10.566 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T21:35:10.567 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:35:10.567 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-22T21:35:10.567 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:35:10.567 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:35:10.567 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-22 21:34:55.734000414 +0000 2024-04-22T21:35:10.567 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-22 21:32:28.326830024 +0000 2024-04-22T21:35:10.567 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-22 21:32:28.326830024 +0000 2024-04-22T21:35:10.567 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-22 21:32:28.326830024 +0000 2024-04-22T21:35:10.567 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T21:35:10.633 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-22T21:35:10.634 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-22T21:35:10.634 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000264716 s, 1.9 MB/s 2024-04-22T21:35:10.635 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T21:35:10.694 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-04-22T21:35:10.750 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T21:35:10.750 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:35:10.750 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-22T21:35:10.750 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:35:10.750 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:35:10.750 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-22 21:34:55.734000414 +0000 2024-04-22T21:35:10.751 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-22 21:32:28.655824189 +0000 2024-04-22T21:35:10.751 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-22 21:32:28.655824189 +0000 2024-04-22T21:35:10.751 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-22 21:32:28.655824189 +0000 2024-04-22T21:35:10.751 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T21:35:10.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:10 smithi165 ceph-mon[26549]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi165=y;smithi188=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:10.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:10 smithi165 ceph-mon[26549]: Saving service mgr spec with placement smithi165=y;smithi188=x;count:2 2024-04-22T21:35:10.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:10 smithi165 ceph-mon[21833]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi165=y;smithi188=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:10.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:10 smithi165 ceph-mon[21833]: Saving service mgr spec with placement smithi165=y;smithi188=x;count:2 2024-04-22T21:35:10.815 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-22T21:35:10.815 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-22T21:35:10.815 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000188247 s, 2.7 MB/s 2024-04-22T21:35:10.816 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T21:35:10.874 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-04-22T21:35:10.931 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T21:35:10.931 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:35:10.931 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-22T21:35:10.931 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:35:10.931 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:35:10.931 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-22 21:34:55.735000397 +0000 2024-04-22T21:35:10.931 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-22 21:32:29.005817982 +0000 2024-04-22T21:35:10.931 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-22 21:32:29.005817982 +0000 2024-04-22T21:35:10.931 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-22 21:32:29.005817982 +0000 2024-04-22T21:35:10.931 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T21:35:10.991 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:10 smithi188 ceph-mon[22698]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi165=y;smithi188=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:10.991 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:10 smithi188 ceph-mon[22698]: Saving service mgr spec with placement smithi165=y;smithi188=x;count:2 2024-04-22T21:35:10.993 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-22T21:35:10.993 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-22T21:35:10.993 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000176563 s, 2.9 MB/s 2024-04-22T21:35:10.994 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T21:35:11.050 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-04-22T21:35:11.105 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T21:35:11.105 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T21:35:11.105 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-22T21:35:11.105 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T21:35:11.105 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T21:35:11.105 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-22 21:34:55.735000397 +0000 2024-04-22T21:35:11.105 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-22 21:32:29.352811828 +0000 2024-04-22T21:35:11.105 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-22 21:32:29.352811828 +0000 2024-04-22T21:35:11.105 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-22 21:32:29.352811828 +0000 2024-04-22T21:35:11.106 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T21:35:11.167 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-22T21:35:11.167 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-22T21:35:11.167 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000211998 s, 2.4 MB/s 2024-04-22T21:35:11.168 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T21:35:11.224 INFO:tasks.cephadm:Deploying osd.0 on smithi165 with /dev/vg_nvme/lv_4... 2024-04-22T21:35:11.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T21:35:11.780 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:11.780 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:11.780 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:11.780 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:11.780 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[26549]: Deploying daemon mgr.x on smithi188 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:35:11.781 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T21:35:11.782 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:35:11.782 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:11.782 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:11 smithi165 ceph-mon[21833]: Deploying daemon mgr.x on smithi188 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:11.995 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:11 smithi188 ceph-mon[22698]: Deploying daemon mgr.x on smithi188 2024-04-22T21:35:13.297 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T21:35:13.297 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T21:35:13.297 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-04-22T21:35:13.297 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-04-22T21:35:13.297 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135324 s, 775 MB/s 2024-04-22T21:35:13.297 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-04-22T21:35:13.494 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:13 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:13.182+0000 7fbb5d05c000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T21:35:13.494 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:13 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:13.510+0000 7fbb5d05c000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T21:35:13.758 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:13 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:13.596+0000 7fbb5d05c000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T21:35:13.966 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[21833]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:13.966 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:13.966 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:13.966 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:13.966 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:13.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[26549]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:13.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:13.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:13.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:13.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:13 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:13.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_4 2024-04-22T21:35:14.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:13 smithi188 ceph-mon[22698]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:14.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:13 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:14.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:13 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:14.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:13 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:14.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:13 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:15.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:15.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:15.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-mon[22698]: Reconfiguring mgr.y (unknown last config time)... 2024-04-22T21:35:15.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:35:15.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:35:15.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:15.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-mon[22698]: Reconfiguring daemon mgr.y on smithi165 2024-04-22T21:35:15.008 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:14.727+0000 7fbb5d05c000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T21:35:15.009 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:14.886+0000 7fbb5d05c000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T21:35:15.009 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:14.966+0000 7fbb5d05c000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T21:35:15.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:15.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:15.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[21833]: Reconfiguring mgr.y (unknown last config time)... 2024-04-22T21:35:15.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:35:15.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:35:15.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:15.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[21833]: Reconfiguring daemon mgr.y on smithi165 2024-04-22T21:35:15.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:15.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:15.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[26549]: Reconfiguring mgr.y (unknown last config time)... 2024-04-22T21:35:15.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T21:35:15.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T21:35:15.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:15.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:14 smithi165 ceph-mon[26549]: Reconfiguring daemon mgr.y on smithi165 2024-04-22T21:35:15.508 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:15.272+0000 7fbb5d05c000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T21:35:15.508 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:15.359+0000 7fbb5d05c000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T21:35:15.508 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:15.449+0000 7fbb5d05c000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T21:35:15.966 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:15.686+0000 7fbb5d05c000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:35:16.240 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:15 smithi165 ceph-mon[21833]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:16.241 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:15 smithi165 ceph-mon[26549]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:16.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:15 smithi188 ceph-mon[22698]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:16.258 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:16.114+0000 7fbb5d05c000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T21:35:16.949 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:16.589+0000 7fbb5d05c000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T21:35:16.949 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:16.679+0000 7fbb5d05c000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T21:35:16.949 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:16.965+0000 7fbb5d05c000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T21:35:17.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:35:17.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:17.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:17.060 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:17.060 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:17.258 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:17.055+0000 7fbb5d05c000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T21:35:17.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:35:17.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:35:17.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:17.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:17.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:17.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:17.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:17.259 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:17.758 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:17.452+0000 7fbb5d05c000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T21:35:17.758 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:17.579+0000 7fbb5d05c000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T21:35:17.758 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:17.669+0000 7fbb5d05c000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T21:35:18.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:17 smithi165 ceph-mon[21833]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:18.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:35:18.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:35:18.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:18.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:17 smithi165 ceph-mon[26549]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:18.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:17 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:35:18.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:17 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:35:18.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:17 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:18.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-mon[22698]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:18.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:35:18.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:35:18.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:18.258 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:17.780+0000 7fbb5d05c000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T21:35:18.258 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:17.866+0000 7fbb5d05c000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T21:35:18.738 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:18 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:18.400+0000 7fbb5d05c000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T21:35:19.008 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:35:18 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:35:18.755+0000 7fbb5d05c000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T21:35:19.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:18 smithi188 ceph-mon[22698]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:19.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:18 smithi188 ceph-mon[22698]: Standby manager daemon x started 2024-04-22T21:35:19.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:18 smithi188 ceph-mon[22698]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T21:35:19.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:18 smithi188 ceph-mon[22698]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T21:35:19.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:18 smithi188 ceph-mon[22698]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T21:35:19.009 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:18 smithi188 ceph-mon[22698]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[21833]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[21833]: Standby manager daemon x started 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[21833]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[21833]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[21833]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[21833]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[26549]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[26549]: Standby manager daemon x started 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[26549]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T21:35:19.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[26549]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T21:35:19.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[26549]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T21:35:19.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:18 smithi165 ceph-mon[26549]: from='mgr.? 172.21.15.188:0/962802656' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T21:35:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:19 smithi188 ceph-mon[22698]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:19 smithi188 ceph-mon[22698]: mgrmap e14: y(active, since 62s), standbys: x 2024-04-22T21:35:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:19 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T21:35:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:19 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3400222537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315d5f25-fb61-4586-b222-e4463c47ac34"}]: dispatch 2024-04-22T21:35:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:19 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3400222537' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315d5f25-fb61-4586-b222-e4463c47ac34"}]': finished 2024-04-22T21:35:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:19 smithi188 ceph-mon[22698]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T21:35:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:19 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:20.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[21833]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:20.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[21833]: mgrmap e14: y(active, since 62s), standbys: x 2024-04-22T21:35:20.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T21:35:20.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3400222537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315d5f25-fb61-4586-b222-e4463c47ac34"}]: dispatch 2024-04-22T21:35:20.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3400222537' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315d5f25-fb61-4586-b222-e4463c47ac34"}]': finished 2024-04-22T21:35:20.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[21833]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T21:35:20.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:20.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[26549]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:20.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[26549]: mgrmap e14: y(active, since 62s), standbys: x 2024-04-22T21:35:20.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T21:35:20.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3400222537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315d5f25-fb61-4586-b222-e4463c47ac34"}]: dispatch 2024-04-22T21:35:20.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3400222537' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315d5f25-fb61-4586-b222-e4463c47ac34"}]': finished 2024-04-22T21:35:20.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[26549]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T21:35:20.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:19 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:21.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:21 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3086422914' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:35:21.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:21 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3086422914' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:35:21.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:21 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3086422914' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:35:22.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:22 smithi188 ceph-mon[22698]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:22.267 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:22 smithi165 ceph-mon[21833]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:22.267 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:22 smithi165 ceph-mon[26549]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:24.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:24 smithi188 ceph-mon[22698]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:24.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:24 smithi165 ceph-mon[26549]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:24.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:24 smithi165 ceph-mon[21833]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:26 smithi188 ceph-mon[22698]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:26 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T21:35:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:26 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:26 smithi165 ceph-mon[21833]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:26 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T21:35:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:26 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:26.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:26 smithi165 ceph-mon[26549]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:26.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:26 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T21:35:26.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:26 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:27.035 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:27 smithi165 ceph-mon[21833]: Deploying daemon osd.0 on smithi165 2024-04-22T21:35:27.035 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:27 smithi165 ceph-mon[26549]: Deploying daemon osd.0 on smithi165 2024-04-22T21:35:27.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:27 smithi188 ceph-mon[22698]: Deploying daemon osd.0 on smithi165 2024-04-22T21:35:28.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:28 smithi188 ceph-mon[22698]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:28.292 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:28 smithi165 ceph-mon[21833]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:28.292 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:28 smithi165 ceph-mon[26549]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:30.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:30 smithi188 ceph-mon[22698]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:30.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:30 smithi165 ceph-mon[21833]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:30.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:30 smithi165 ceph-mon[26549]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:32.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:31 smithi188 ceph-mon[22698]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:32.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:32.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:32.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:32.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:32.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[21833]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:32.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:32.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:32.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:32.010 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:32.011 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[26549]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:32.011 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:32.011 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:32.011 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:32.011 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:33.786 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 0 on host 'smithi165' 2024-04-22T21:35:34.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:33 smithi188 ceph-mon[22698]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:34.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:33 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:34.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:33 smithi165 ceph-mon[21833]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:34.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:33 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:34.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:33 smithi165 ceph-mon[26549]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:34.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:33 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:34.432 DEBUG:teuthology.orchestra.run.smithi165:osd.0> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T21:35:34.434 INFO:tasks.cephadm:Deploying osd.1 on smithi165 with /dev/vg_nvme/lv_3... 2024-04-22T21:35:34.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T21:35:34.981 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:34 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:34.981 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:34 smithi165 ceph-mon[21833]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T21:35:34.981 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:34 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:34.981 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:34 smithi165 ceph-mon[26549]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T21:35:35.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:34 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:35.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:34 smithi188 ceph-mon[22698]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T21:35:36.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[26549]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:36.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[26549]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T21:35:36.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[26549]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T21:35:36.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:36.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[26549]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:35:36.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[21833]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:36.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[21833]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T21:35:36.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[21833]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T21:35:36.060 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:36.060 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-mon[21833]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:35:36.060 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0[29420]: 2024-04-22T21:35:35.814+0000 7f6119307700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T21:35:36.060 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:35:35 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0[29420]: 2024-04-22T21:35:35.817+0000 7f6111c9a700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:35:36.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:35 smithi188 ceph-mon[22698]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:36.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:35 smithi188 ceph-mon[22698]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T21:35:36.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:35 smithi188 ceph-mon[22698]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T21:35:36.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:35 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:36.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:35 smithi188 ceph-mon[22698]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:35:37.037 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T21:35:37.037 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T21:35:37.037 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-04-22T21:35:37.037 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-04-22T21:35:37.037 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012394 s, 846 MB/s 2024-04-22T21:35:37.037 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-04-22T21:35:37.037 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:36 smithi165 ceph-mon[26549]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:35:37.037 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:36 smithi165 ceph-mon[26549]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T21:35:37.037 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:36 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:37.038 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:36 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:37.038 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:36 smithi165 ceph-mon[21833]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:35:37.038 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:36 smithi165 ceph-mon[21833]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T21:35:37.038 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:36 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:37.038 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:36 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:37.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:36 smithi188 ceph-mon[22698]: from='osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:35:37.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:36 smithi188 ceph-mon[22698]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T21:35:37.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:36 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:37.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:36 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:37.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_3 2024-04-22T21:35:38.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:37 smithi188 ceph-mon[22698]: purged_snaps scrub starts 2024-04-22T21:35:38.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:37 smithi188 ceph-mon[22698]: purged_snaps scrub ok 2024-04-22T21:35:38.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:37 smithi188 ceph-mon[22698]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:38.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:37 smithi188 ceph-mon[22698]: osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935] boot 2024-04-22T21:35:38.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:37 smithi188 ceph-mon[22698]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T21:35:38.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:37 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:38.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[21833]: purged_snaps scrub starts 2024-04-22T21:35:38.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[21833]: purged_snaps scrub ok 2024-04-22T21:35:38.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[21833]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:38.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[21833]: osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935] boot 2024-04-22T21:35:38.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[21833]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T21:35:38.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:38.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[26549]: purged_snaps scrub starts 2024-04-22T21:35:38.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[26549]: purged_snaps scrub ok 2024-04-22T21:35:38.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[26549]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T21:35:38.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[26549]: osd.0 [v2:172.21.15.165:6802/3819369935,v1:172.21.15.165:6803/3819369935] boot 2024-04-22T21:35:38.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[26549]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T21:35:38.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:37 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:35:39.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:38 smithi188 ceph-mon[22698]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T21:35:39.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:38 smithi165 ceph-mon[21833]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T21:35:39.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:38 smithi165 ceph-mon[26549]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T21:35:40.019 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:39 smithi165 ceph-mon[21833]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:40.019 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:39 smithi165 ceph-mon[26549]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:40.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:39 smithi188 ceph-mon[22698]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:40.928 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:40.929 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:40.930 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:40 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:41.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:41.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:40 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:41.942 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:41 smithi165 ceph-mon[21833]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:41.942 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:41 smithi165 ceph-mon[21833]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:41.942 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:41 smithi165 ceph-mon[21833]: Detected new or changed devices on smithi165 2024-04-22T21:35:41.943 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:41 smithi165 ceph-mon[21833]: Adjusting osd_memory_target on smithi165 to 16064M 2024-04-22T21:35:41.943 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:41 smithi165 ceph-mon[26549]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:41.943 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:41 smithi165 ceph-mon[26549]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:41.943 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:41 smithi165 ceph-mon[26549]: Detected new or changed devices on smithi165 2024-04-22T21:35:41.943 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:41 smithi165 ceph-mon[26549]: Adjusting osd_memory_target on smithi165 to 16064M 2024-04-22T21:35:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:41 smithi188 ceph-mon[22698]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:41 smithi188 ceph-mon[22698]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:35:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:41 smithi188 ceph-mon[22698]: Detected new or changed devices on smithi165 2024-04-22T21:35:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:41 smithi188 ceph-mon[22698]: Adjusting osd_memory_target on smithi165 to 16064M 2024-04-22T21:35:43.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[21833]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:43.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/95011328' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e61383e-0138-4fb4-ac5a-1a9f96406576"}]: dispatch 2024-04-22T21:35:43.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/95011328' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e61383e-0138-4fb4-ac5a-1a9f96406576"}]': finished 2024-04-22T21:35:43.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[21833]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T21:35:43.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:35:43.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1273879885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:35:43.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[26549]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:43.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/95011328' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e61383e-0138-4fb4-ac5a-1a9f96406576"}]: dispatch 2024-04-22T21:35:43.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/95011328' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e61383e-0138-4fb4-ac5a-1a9f96406576"}]': finished 2024-04-22T21:35:43.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[26549]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T21:35:43.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:35:43.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:43 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1273879885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:35:43.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:43 smithi188 ceph-mon[22698]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:43.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:43 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/95011328' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e61383e-0138-4fb4-ac5a-1a9f96406576"}]: dispatch 2024-04-22T21:35:43.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:43 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/95011328' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e61383e-0138-4fb4-ac5a-1a9f96406576"}]': finished 2024-04-22T21:35:43.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:43 smithi188 ceph-mon[22698]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T21:35:43.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:43 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:35:43.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:43 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1273879885' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:35:45.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:45 smithi165 ceph-mon[21833]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:45.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:45 smithi165 ceph-mon[26549]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:45.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:45 smithi188 ceph-mon[22698]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:47.516 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:47 smithi165 ceph-mon[21833]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:47.516 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:47 smithi165 ceph-mon[26549]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:47.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:47 smithi188 ceph-mon[22698]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:49.651 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:49 smithi165 ceph-mon[21833]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:49.651 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:49 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T21:35:49.651 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:49 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:49.651 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:49 smithi165 ceph-mon[21833]: Deploying daemon osd.1 on smithi165 2024-04-22T21:35:49.651 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:49 smithi165 ceph-mon[26549]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:49.651 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:49 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T21:35:49.651 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:49 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:49.652 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:49 smithi165 ceph-mon[26549]: Deploying daemon osd.1 on smithi165 2024-04-22T21:35:49.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:49 smithi188 ceph-mon[22698]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:49.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:49 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T21:35:49.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:49 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:49.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:49 smithi188 ceph-mon[22698]: Deploying daemon osd.1 on smithi165 2024-04-22T21:35:51.601 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:51 smithi165 ceph-mon[21833]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:51.602 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:51 smithi165 ceph-mon[26549]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:51.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:51 smithi188 ceph-mon[22698]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:53.560 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:53 smithi165 ceph-mon[21833]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:53.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:53 smithi165 ceph-mon[26549]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:53.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:53 smithi188 ceph-mon[22698]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:55.245 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[26549]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:55.245 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:55.245 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:55.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:55 smithi188 ceph-mon[22698]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:55.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:55 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:55.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:55 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:55.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:55 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:55.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:55 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:55.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:55.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:55.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[21833]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:55.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:55.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:55.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:55.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:55 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:57.724 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 1 on host 'smithi165' 2024-04-22T21:35:58.022 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[21833]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:58.022 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:58.022 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:58.022 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[21833]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T21:35:58.022 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:58.022 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:58.022 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:58.022 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:58.023 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[26549]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:58.023 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:58.023 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:58.023 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[26549]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T21:35:58.023 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:58.023 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:58.023 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:58.023 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:57 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:58.211 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:57 smithi188 ceph-mon[22698]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:58.211 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:58.211 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:58.211 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:57 smithi188 ceph-mon[22698]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T21:35:58.212 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:35:58.212 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:35:58.212 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:35:58.212 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:35:58.337 DEBUG:teuthology.orchestra.run.smithi165:osd.1> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T21:35:58.339 INFO:tasks.cephadm:Deploying osd.2 on smithi165 with /dev/vg_nvme/lv_2... 2024-04-22T21:35:58.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T21:35:59.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[21833]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:59.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[21833]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T21:35:59.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[21833]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T21:35:59.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:35:59.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[21833]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:35:59.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[26549]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:59.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[26549]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T21:35:59.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[26549]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T21:35:59.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:35:59.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-mon[26549]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:35:59.560 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1[32379]: 2024-04-22T21:35:59.332+0000 7f1c50df3700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T21:35:59.560 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:35:59 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1[32379]: 2024-04-22T21:35:59.335+0000 7f1c4d78e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:35:59.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:59 smithi188 ceph-mon[22698]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:35:59.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:59 smithi188 ceph-mon[22698]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T21:35:59.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:59 smithi188 ceph-mon[22698]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T21:35:59.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:59 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:35:59.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:35:59 smithi188 ceph-mon[22698]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:00.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:00 smithi165 ceph-mon[26549]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:36:00.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:00 smithi165 ceph-mon[26549]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T21:36:00.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:00 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:36:00.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:00 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:36:00.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:00 smithi165 ceph-mon[21833]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:36:00.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:00 smithi165 ceph-mon[21833]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T21:36:00.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:00 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:36:00.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:00 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:36:00.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:00 smithi188 ceph-mon[22698]: from='osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:36:00.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:00 smithi188 ceph-mon[22698]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T21:36:00.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:00 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:36:00.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:00 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:36:01.422 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T21:36:01.422 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T21:36:01.422 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-04-22T21:36:01.422 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-04-22T21:36:01.422 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013067 s, 802 MB/s 2024-04-22T21:36:01.422 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-04-22T21:36:01.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[21833]: purged_snaps scrub starts 2024-04-22T21:36:01.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[21833]: purged_snaps scrub ok 2024-04-22T21:36:01.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[21833]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:36:01.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[21833]: osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941] boot 2024-04-22T21:36:01.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[21833]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T21:36:01.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:36:01.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[26549]: purged_snaps scrub starts 2024-04-22T21:36:01.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[26549]: purged_snaps scrub ok 2024-04-22T21:36:01.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[26549]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:36:01.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[26549]: osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941] boot 2024-04-22T21:36:01.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[26549]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T21:36:01.560 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:36:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:01 smithi188 ceph-mon[22698]: purged_snaps scrub starts 2024-04-22T21:36:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:01 smithi188 ceph-mon[22698]: purged_snaps scrub ok 2024-04-22T21:36:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:01 smithi188 ceph-mon[22698]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T21:36:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:01 smithi188 ceph-mon[22698]: osd.1 [v2:172.21.15.165:6810/4196345941,v1:172.21.15.165:6811/4196345941] boot 2024-04-22T21:36:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:01 smithi188 ceph-mon[22698]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T21:36:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:36:02.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_2 2024-04-22T21:36:02.514 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:02 smithi165 ceph-mon[21833]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T21:36:02.514 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:02 smithi165 ceph-mon[26549]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T21:36:02.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:02 smithi188 ceph-mon[22698]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T21:36:04.185 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[21833]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[21833]: Detected new or changed devices on smithi165 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[21833]: Adjusting osd_memory_target on smithi165 to 8032M 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[26549]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[26549]: Detected new or changed devices on smithi165 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:04.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:04.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[26549]: Adjusting osd_memory_target on smithi165 to 8032M 2024-04-22T21:36:04.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:04.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:03 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:04.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:03 smithi188 ceph-mon[22698]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:04.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:03 smithi188 ceph-mon[22698]: Detected new or changed devices on smithi165 2024-04-22T21:36:04.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:03 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:04.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:03 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:04.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:03 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:04.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:03 smithi188 ceph-mon[22698]: Adjusting osd_memory_target on smithi165 to 8032M 2024-04-22T21:36:04.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:03 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:04.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:03 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:05.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:04 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:36:05.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:04 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:36:05.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:04 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:05.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:04 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:36:05.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:04 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:36:05.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:04 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:05.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:36:05.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:36:05.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:04 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:06.098 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:05 smithi165 ceph-mon[21833]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:06.098 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:05 smithi165 ceph-mon[21833]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:06.099 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:05 smithi165 ceph-mon[26549]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:06.099 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:05 smithi165 ceph-mon[26549]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:06.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:05 smithi188 ceph-mon[22698]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:06.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:05 smithi188 ceph-mon[22698]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:08.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:07 smithi188 ceph-mon[22698]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:08.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:07 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3534372844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1b573c-17c0-4e85-9f32-e66637e8fe0a"}]: dispatch 2024-04-22T21:36:08.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:07 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1b573c-17c0-4e85-9f32-e66637e8fe0a"}]: dispatch 2024-04-22T21:36:08.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:07 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b1b573c-17c0-4e85-9f32-e66637e8fe0a"}]': finished 2024-04-22T21:36:08.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:07 smithi188 ceph-mon[22698]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T21:36:08.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:07 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:08.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:07 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/770740093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[21833]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3534372844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1b573c-17c0-4e85-9f32-e66637e8fe0a"}]: dispatch 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1b573c-17c0-4e85-9f32-e66637e8fe0a"}]: dispatch 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b1b573c-17c0-4e85-9f32-e66637e8fe0a"}]': finished 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[21833]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/770740093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[26549]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3534372844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1b573c-17c0-4e85-9f32-e66637e8fe0a"}]: dispatch 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b1b573c-17c0-4e85-9f32-e66637e8fe0a"}]: dispatch 2024-04-22T21:36:08.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b1b573c-17c0-4e85-9f32-e66637e8fe0a"}]': finished 2024-04-22T21:36:08.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[26549]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T21:36:08.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:08.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:07 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/770740093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:36:10.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:09 smithi188 ceph-mon[22698]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:10.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:09 smithi165 ceph-mon[21833]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:10.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:09 smithi165 ceph-mon[26549]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:12.026 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:11 smithi165 ceph-mon[21833]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:12.026 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:11 smithi165 ceph-mon[26549]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:12.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:11 smithi188 ceph-mon[22698]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:13 smithi165 ceph-mon[21833]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:13 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T21:36:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:13 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:14.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:13 smithi165 ceph-mon[26549]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:14.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:13 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T21:36:14.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:13 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:14.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:13 smithi188 ceph-mon[22698]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:14.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:13 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T21:36:14.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:13 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:14.987 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:14 smithi165 ceph-mon[21833]: Deploying daemon osd.2 on smithi165 2024-04-22T21:36:14.987 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:14 smithi165 ceph-mon[26549]: Deploying daemon osd.2 on smithi165 2024-04-22T21:36:15.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:14 smithi188 ceph-mon[22698]: Deploying daemon osd.2 on smithi165 2024-04-22T21:36:15.995 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:15 smithi165 ceph-mon[21833]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:15.996 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:15 smithi165 ceph-mon[26549]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:16.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:15 smithi188 ceph-mon[22698]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:17.019 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:36:17.019 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:36:17.019 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:36:17.019 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:36:17.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:36:17.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:36:18.185 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:17 smithi165 ceph-mon[21833]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:18.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:17 smithi165 ceph-mon[26549]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:18.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:17 smithi188 ceph-mon[22698]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[26549]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[21833]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:20.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:19 smithi188 ceph-mon[22698]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:19 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:19 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:19 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:20.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:19 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:21.903 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:21 smithi165 ceph-mon[26549]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:21.903 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:21 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:21.903 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:21 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:21.903 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:21 smithi165 ceph-mon[21833]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:21.903 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:21 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:21.903 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:21 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:22.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:21 smithi188 ceph-mon[22698]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:22.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:21 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:22.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:21 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:22.437 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 2 on host 'smithi165' 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[26549]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[21833]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:23.075 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:22 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:23.097 DEBUG:teuthology.orchestra.run.smithi165:osd.2> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T21:36:23.099 INFO:tasks.cephadm:Deploying osd.3 on smithi165 with /dev/vg_nvme/lv_1... 2024-04-22T21:36:23.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T21:36:23.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:22 smithi188 ceph-mon[22698]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T21:36:23.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:22 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:23.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:22 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:23.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:22 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:23.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:22 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:24.106 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[21833]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:24.106 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[21833]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T21:36:24.106 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[21833]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T21:36:24.106 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:24.106 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[21833]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:24.107 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[26549]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:24.107 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[26549]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T21:36:24.107 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[26549]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T21:36:24.107 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:24.107 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-mon[26549]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:24.107 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2[35305]: 2024-04-22T21:36:23.834+0000 7f8606811700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T21:36:24.107 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:36:23 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2[35305]: 2024-04-22T21:36:23.837+0000 7f86019a9700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:36:24.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:23 smithi188 ceph-mon[22698]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:24.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:23 smithi188 ceph-mon[22698]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T21:36:24.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:23 smithi188 ceph-mon[22698]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T21:36:24.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:24.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:23 smithi188 ceph-mon[22698]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:25.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:24 smithi165 ceph-mon[21833]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:36:25.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:24 smithi165 ceph-mon[21833]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T21:36:25.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:24 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:25.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:24 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:25.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:24 smithi165 ceph-mon[26549]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:36:25.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:24 smithi165 ceph-mon[26549]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T21:36:25.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:24 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:25.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:24 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:25.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:24 smithi188 ceph-mon[22698]: from='osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:36:25.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:24 smithi188 ceph-mon[22698]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T21:36:25.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:24 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:25.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:24 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:26.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[21833]: purged_snaps scrub starts 2024-04-22T21:36:26.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[21833]: purged_snaps scrub ok 2024-04-22T21:36:26.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[21833]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:26.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:26.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[21833]: osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826] boot 2024-04-22T21:36:26.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[21833]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T21:36:26.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:26.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[26549]: purged_snaps scrub starts 2024-04-22T21:36:26.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[26549]: purged_snaps scrub ok 2024-04-22T21:36:26.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[26549]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:26.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:26.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[26549]: osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826] boot 2024-04-22T21:36:26.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[26549]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T21:36:26.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:25 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:25 smithi188 ceph-mon[22698]: purged_snaps scrub starts 2024-04-22T21:36:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:25 smithi188 ceph-mon[22698]: purged_snaps scrub ok 2024-04-22T21:36:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:25 smithi188 ceph-mon[22698]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T21:36:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:25 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:25 smithi188 ceph-mon[22698]: osd.2 [v2:172.21.15.165:6818/4051532826,v1:172.21.15.165:6819/4051532826] boot 2024-04-22T21:36:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:25 smithi188 ceph-mon[22698]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T21:36:26.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:25 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:36:26.750 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T21:36:26.750 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T21:36:26.750 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-04-22T21:36:26.751 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-04-22T21:36:26.751 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123454 s, 849 MB/s 2024-04-22T21:36:26.751 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-04-22T21:36:27.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:26 smithi165 ceph-mon[21833]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T21:36:27.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:26 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T21:36:27.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:26 smithi165 ceph-mon[26549]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T21:36:27.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:26 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T21:36:27.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:26 smithi188 ceph-mon[22698]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T21:36:27.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:26 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T21:36:27.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_1 2024-04-22T21:36:28.159 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:27 smithi188 ceph-mon[22698]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:28.159 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:27 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T21:36:28.159 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:27 smithi188 ceph-mon[22698]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T21:36:28.159 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:27 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T21:36:28.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:27 smithi165 ceph-mon[21833]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:28.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:27 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T21:36:28.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:27 smithi165 ceph-mon[21833]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T21:36:28.186 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:27 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T21:36:28.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:27 smithi165 ceph-mon[26549]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:28.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:27 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T21:36:28.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:27 smithi165 ceph-mon[26549]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T21:36:28.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:27 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T21:36:28.741 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:36:28 smithi165 sudo[36603]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:36:28.742 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:36:28 smithi165 sudo[36603]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:28.742 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:36:28 smithi165 sudo[36603]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:29.012 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:36:28 smithi165 sudo[36608]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:36:29.012 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:36:28 smithi165 sudo[36608]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:29.012 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:36:28 smithi165 sudo[36608]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:29.012 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T21:36:29.012 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T21:36:29.012 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: Detected new or changed devices on smithi165 2024-04-22T21:36:29.012 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.012 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:29.012 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: Adjusting osd_memory_target on smithi165 to 5354M 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: Detected new or changed devices on smithi165 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: Adjusting osd_memory_target on smithi165 to 5354M 2024-04-22T21:36:29.013 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.014 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.014 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:28 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: Detected new or changed devices on smithi165 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: Adjusting osd_memory_target on smithi165 to 5354M 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.259 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:28 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:29.274 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36611]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:36:29.274 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36611]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:29.274 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36611]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:29.535 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36632]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:36:29.536 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36632]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:29.536 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36632]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:29.536 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36639]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:36:29.536 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36639]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:29.536 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36639]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:29.808 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36665]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:36:29.808 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36665]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:29.808 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:36:29 smithi165 sudo[36665]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:30.283 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36669]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:36:30.283 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36669]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:30.283 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36669]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:30.545 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:30 smithi165 ceph-mon[26549]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:30.545 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:30 smithi165 ceph-mon[26549]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T21:36:30.545 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:30 smithi165 ceph-mon[26549]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:36:30.546 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36672]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:36:30.546 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36672]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:30.546 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36672]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:30.546 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:30 smithi165 ceph-mon[21833]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:30.546 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:30 smithi165 ceph-mon[21833]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T21:36:30.546 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:30 smithi165 ceph-mon[21833]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:36:30.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:30 smithi188 ceph-mon[22698]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:30.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:30 smithi188 ceph-mon[22698]: osdmap e22: 3 total, 3 up, 3 in 2024-04-22T21:36:30.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:30 smithi188 ceph-mon[22698]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:36:30.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36739]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:36:30.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36739]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:30.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36739]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:31.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36759]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:36:31.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36759]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:31.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:30 smithi165 sudo[36759]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:31.302 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 sudo[24115]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T21:36:31.302 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 sudo[24115]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:31.302 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 sudo[24115]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:31.642 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:36:31.643 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:31.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:36:31.645 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:36:31.645 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:36:31.645 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:31 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 sudo[24118]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 sudo[24118]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 sudo[24118]: pam_unix(sudo:session): session closed for user root 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:36:31.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:36:31.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:36:31.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:36:31.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:36:31.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:36:31.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:31.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:36:31.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:36:31.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:36:31.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:31 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:36:32.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:32 smithi188 ceph-mon[22698]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:36:32.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:32 smithi188 ceph-mon[22698]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:36:32.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:32 smithi188 ceph-mon[22698]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-22T21:36:32.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:32 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:32.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:32 smithi165 ceph-mon[21833]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:36:32.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:32 smithi165 ceph-mon[21833]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:36:32.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:32 smithi165 ceph-mon[21833]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-22T21:36:32.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:32.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:32 smithi165 ceph-mon[26549]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T21:36:32.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:32 smithi165 ceph-mon[26549]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T21:36:32.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:32 smithi165 ceph-mon[26549]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-22T21:36:32.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:32 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:33.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:33 smithi188 ceph-mon[22698]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:33.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:33 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1477149164' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6e8740-aefe-4aa7-a6bc-45a8fe626b24"}]: dispatch 2024-04-22T21:36:33.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:33 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1477149164' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a6e8740-aefe-4aa7-a6bc-45a8fe626b24"}]': finished 2024-04-22T21:36:33.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:33 smithi188 ceph-mon[22698]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T21:36:33.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:33 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:33.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:33 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1088986308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:36:33.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[21833]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:33.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1477149164' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6e8740-aefe-4aa7-a6bc-45a8fe626b24"}]: dispatch 2024-04-22T21:36:33.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1477149164' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a6e8740-aefe-4aa7-a6bc-45a8fe626b24"}]': finished 2024-04-22T21:36:33.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[21833]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T21:36:33.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:33.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1088986308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:36:33.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[26549]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:33.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1477149164' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6e8740-aefe-4aa7-a6bc-45a8fe626b24"}]: dispatch 2024-04-22T21:36:33.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1477149164' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a6e8740-aefe-4aa7-a6bc-45a8fe626b24"}]': finished 2024-04-22T21:36:33.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[26549]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T21:36:33.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:33.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:33 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1088986308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:36:35.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:35 smithi188 ceph-mon[22698]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:35.768 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:35 smithi165 ceph-mon[21833]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:35.769 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:35 smithi165 ceph-mon[26549]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:37.408 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:37 smithi165 ceph-mon[21833]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:37.409 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:37 smithi165 ceph-mon[26549]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:37.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:37 smithi188 ceph-mon[22698]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:39.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:39 smithi165 ceph-mon[21833]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 86 KiB/s wr, 5 op/s 2024-04-22T21:36:39.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:39 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T21:36:39.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:39 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:39.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:39 smithi165 ceph-mon[21833]: Deploying daemon osd.3 on smithi165 2024-04-22T21:36:39.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:39 smithi165 ceph-mon[26549]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 86 KiB/s wr, 5 op/s 2024-04-22T21:36:39.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:39 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T21:36:39.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:39 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:39.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:39 smithi165 ceph-mon[26549]: Deploying daemon osd.3 on smithi165 2024-04-22T21:36:39.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:39 smithi188 ceph-mon[22698]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 86 KiB/s wr, 5 op/s 2024-04-22T21:36:39.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:39 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T21:36:39.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:39 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:39.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:39 smithi188 ceph-mon[22698]: Deploying daemon osd.3 on smithi165 2024-04-22T21:36:41.510 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:41 smithi165 ceph-mon[21833]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T21:36:41.510 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:41 smithi165 ceph-mon[26549]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T21:36:41.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:41 smithi188 ceph-mon[22698]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-22T21:36:43.443 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:43 smithi165 ceph-mon[21833]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 52 KiB/s wr, 3 op/s 2024-04-22T21:36:43.443 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:43 smithi165 ceph-mon[26549]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 52 KiB/s wr, 3 op/s 2024-04-22T21:36:43.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:43 smithi188 ceph-mon[22698]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 52 KiB/s wr, 3 op/s 2024-04-22T21:36:45.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:45 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:45.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:45 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:45.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:45 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:45.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:45 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:45.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:45 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:45.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:45 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:45.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:45 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:45.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:45 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:45.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:45 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:45.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:45 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:45.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:45 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:45.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:45 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:46.302 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:46 smithi165 ceph-mon[21833]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 975 B/s rd, 45 KiB/s wr, 2 op/s 2024-04-22T21:36:46.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:46 smithi165 ceph-mon[26549]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 975 B/s rd, 45 KiB/s wr, 2 op/s 2024-04-22T21:36:46.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:46 smithi188 ceph-mon[22698]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 975 B/s rd, 45 KiB/s wr, 2 op/s 2024-04-22T21:36:47.432 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 3 on host 'smithi165' 2024-04-22T21:36:48.059 DEBUG:teuthology.orchestra.run.smithi165:osd.3> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T21:36:48.061 INFO:tasks.cephadm:Deploying osd.4 on smithi188 with /dev/vg_nvme/lv_4... 2024-04-22T21:36:48.061 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T21:36:48.297 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:48 smithi165 ceph-mon[21833]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 938 B/s rd, 43 KiB/s wr, 2 op/s 2024-04-22T21:36:48.297 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:48 smithi165 ceph-mon[21833]: from='osd.3 [v2:172.21.15.165:6826/1668061591,v1:172.21.15.165:6827/1668061591]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:36:48.297 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:48 smithi165 ceph-mon[21833]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:36:48.297 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:48 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:48.297 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:48 smithi165 ceph-mon[26549]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 938 B/s rd, 43 KiB/s wr, 2 op/s 2024-04-22T21:36:48.297 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:48 smithi165 ceph-mon[26549]: from='osd.3 [v2:172.21.15.165:6826/1668061591,v1:172.21.15.165:6827/1668061591]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:36:48.297 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:48 smithi165 ceph-mon[26549]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:36:48.297 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:48 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:48.369 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:48 smithi188 ceph-mon[22698]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 938 B/s rd, 43 KiB/s wr, 2 op/s 2024-04-22T21:36:48.369 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:48 smithi188 ceph-mon[22698]: from='osd.3 [v2:172.21.15.165:6826/1668061591,v1:172.21.15.165:6827/1668061591]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:36:48.369 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:48 smithi188 ceph-mon[22698]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T21:36:48.369 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:48 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:49.197 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[21833]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T21:36:49.197 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[21833]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T21:36:49.198 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:49.198 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[21833]: from='osd.3 [v2:172.21.15.165:6826/1668061591,v1:172.21.15.165:6827/1668061591]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:49.198 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[21833]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:49.198 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:49.198 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[26549]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T21:36:49.198 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[26549]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T21:36:49.198 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:49.198 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[26549]: from='osd.3 [v2:172.21.15.165:6826/1668061591,v1:172.21.15.165:6827/1668061591]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:49.199 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[26549]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:49.199 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:49.199 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3[38286]: 2024-04-22T21:36:49.060+0000 7f2ce87e8700 -1 osd.3 0 waiting for initial osdmap 2024-04-22T21:36:49.199 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:36:49 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3[38286]: 2024-04-22T21:36:49.063+0000 7f2ce3980700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:36:49.391 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:49 smithi188 ceph-mon[22698]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T21:36:49.391 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:49 smithi188 ceph-mon[22698]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T21:36:49.391 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:49 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:49.391 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:49 smithi188 ceph-mon[22698]: from='osd.3 [v2:172.21.15.165:6826/1668061591,v1:172.21.15.165:6827/1668061591]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:49.392 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:49 smithi188 ceph-mon[22698]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-04-22T21:36:49.392 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:49 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:50.156 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T21:36:50.156 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T21:36:50.156 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-22T21:36:50.156 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-22T21:36:50.156 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011381 s, 921 MB/s 2024-04-22T21:36:50.156 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[21833]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[21833]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[21833]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[26549]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[26549]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[26549]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:50.202 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:50 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:50.421 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:50 smithi188 ceph-mon[22698]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:50.421 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:50 smithi188 ceph-mon[22698]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-04-22T21:36:50.421 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:50 smithi188 ceph-mon[22698]: osdmap e25: 4 total, 3 up, 4 in 2024-04-22T21:36:50.421 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:50 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:50.421 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:50 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:50.896 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2024-04-22T21:36:51.178 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:51 smithi188 ceph-mon[22698]: purged_snaps scrub starts 2024-04-22T21:36:51.178 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:51 smithi188 ceph-mon[22698]: purged_snaps scrub ok 2024-04-22T21:36:51.178 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:51 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:51.179 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:51 smithi188 ceph-mon[22698]: osd.3 [v2:172.21.15.165:6826/1668061591,v1:172.21.15.165:6827/1668061591] boot 2024-04-22T21:36:51.179 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:51 smithi188 ceph-mon[22698]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T21:36:51.179 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:51 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:51.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[26549]: purged_snaps scrub starts 2024-04-22T21:36:51.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[26549]: purged_snaps scrub ok 2024-04-22T21:36:51.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:51.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[26549]: osd.3 [v2:172.21.15.165:6826/1668061591,v1:172.21.15.165:6827/1668061591] boot 2024-04-22T21:36:51.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[26549]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T21:36:51.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:51.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[21833]: purged_snaps scrub starts 2024-04-22T21:36:51.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[21833]: purged_snaps scrub ok 2024-04-22T21:36:51.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:51.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[21833]: osd.3 [v2:172.21.15.165:6826/1668061591,v1:172.21.15.165:6827/1668061591] boot 2024-04-22T21:36:51.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[21833]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T21:36:51.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:51 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.138 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:52.139 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:52.139 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:52.139 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:52 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: osdmap e27: 4 total, 4 up, 4 in 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:36:52.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:52 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:36:53.142 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:53 smithi188 ceph-mon[22698]: Detected new or changed devices on smithi165 2024-04-22T21:36:53.142 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:53 smithi188 ceph-mon[22698]: Adjusting osd_memory_target on smithi165 to 4016M 2024-04-22T21:36:53.142 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:53 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:36:53.142 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:53 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:36:53.142 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:53 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:53.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[21833]: Detected new or changed devices on smithi165 2024-04-22T21:36:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[21833]: Adjusting osd_memory_target on smithi165 to 4016M 2024-04-22T21:36:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:36:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:36:53.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:53.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[26549]: Detected new or changed devices on smithi165 2024-04-22T21:36:53.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[26549]: Adjusting osd_memory_target on smithi165 to 4016M 2024-04-22T21:36:53.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:36:53.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:36:53.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:53 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:36:54.113 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:54 smithi188 ceph-mon[22698]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:36:54.114 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:54 smithi188 ceph-mon[22698]: from='client.34199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:54.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:54 smithi165 ceph-mon[21833]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:36:54.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:54 smithi165 ceph-mon[21833]: from='client.34199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:54.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:54 smithi165 ceph-mon[26549]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:36:54.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:54 smithi165 ceph-mon[26549]: from='client.34199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:36:56.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:56 smithi188 ceph-mon[22698]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:36:56.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:56 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/284054193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc54e2dc-af18-42ba-897c-b0e06fafb520"}]: dispatch 2024-04-22T21:36:56.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:56 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc54e2dc-af18-42ba-897c-b0e06fafb520"}]: dispatch 2024-04-22T21:36:56.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:56 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc54e2dc-af18-42ba-897c-b0e06fafb520"}]': finished 2024-04-22T21:36:56.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:56 smithi188 ceph-mon[22698]: osdmap e28: 5 total, 4 up, 5 in 2024-04-22T21:36:56.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:56 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:36:56.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:56 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/1309857602' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:36:56.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[21833]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:36:56.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/284054193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc54e2dc-af18-42ba-897c-b0e06fafb520"}]: dispatch 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc54e2dc-af18-42ba-897c-b0e06fafb520"}]: dispatch 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc54e2dc-af18-42ba-897c-b0e06fafb520"}]': finished 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[21833]: osdmap e28: 5 total, 4 up, 5 in 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/1309857602' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[26549]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/284054193' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc54e2dc-af18-42ba-897c-b0e06fafb520"}]: dispatch 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc54e2dc-af18-42ba-897c-b0e06fafb520"}]: dispatch 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc54e2dc-af18-42ba-897c-b0e06fafb520"}]': finished 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[26549]: osdmap e28: 5 total, 4 up, 5 in 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:36:56.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:56 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/1309857602' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:36:58.187 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:36:58 smithi188 ceph-mon[22698]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:36:58.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:36:58 smithi165 ceph-mon[21833]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:36:58.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:36:58 smithi165 ceph-mon[26549]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:00.198 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:00 smithi188 ceph-mon[22698]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:00.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:00 smithi165 ceph-mon[21833]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:00.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:00 smithi165 ceph-mon[26549]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:02.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:02 smithi188 ceph-mon[22698]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:02.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:02 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T21:37:02.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:02 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:02.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:02 smithi165 ceph-mon[21833]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:02.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T21:37:02.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:02 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:02 smithi165 ceph-mon[26549]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T21:37:02.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:02 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:03.148 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:03 smithi188 ceph-mon[22698]: Deploying daemon osd.4 on smithi188 2024-04-22T21:37:03.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:03 smithi165 ceph-mon[21833]: Deploying daemon osd.4 on smithi188 2024-04-22T21:37:03.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:03 smithi165 ceph-mon[26549]: Deploying daemon osd.4 on smithi188 2024-04-22T21:37:04.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:04 smithi188 ceph-mon[22698]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:04.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:04 smithi165 ceph-mon[26549]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:04.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:04 smithi165 ceph-mon[21833]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:05.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:05 smithi188 ceph-mon[22698]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:05.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:05 smithi165 ceph-mon[21833]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:05.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:05 smithi165 ceph-mon[26549]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:07.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:07 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:07.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:07 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:07.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:07 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:07.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:07 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:07.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:07 smithi188 ceph-mon[22698]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:07.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:07.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:07.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:07.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:07.559 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[21833]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:07.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:07.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:07.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:07.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:07.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:07 smithi165 ceph-mon[26549]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:09.532 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 4 on host 'smithi188' 2024-04-22T21:37:10.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:09 smithi188 ceph-mon[22698]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:10.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:09 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:10.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:09 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:10.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:09 smithi188 ceph-mon[22698]: from='osd.4 [v2:172.21.15.188:6800/1586199781,v1:172.21.15.188:6801/1586199781]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:37:10.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:09 smithi188 ceph-mon[22698]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:37:10.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:09 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:10.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:09 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:10.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:09 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:10.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:09 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:10.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[26549]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:10.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:10.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:10.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[26549]: from='osd.4 [v2:172.21.15.188:6800/1586199781,v1:172.21.15.188:6801/1586199781]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[26549]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[21833]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[21833]: from='osd.4 [v2:172.21.15.188:6800/1586199781,v1:172.21.15.188:6801/1586199781]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[21833]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:10.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:10.060 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:09 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:10.168 DEBUG:teuthology.orchestra.run.smithi188:osd.4> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T21:37:10.171 INFO:tasks.cephadm:Deploying osd.5 on smithi188 with /dev/vg_nvme/lv_3... 2024-04-22T21:37:10.171 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T21:37:11.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[21833]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T21:37:11.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[21833]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T21:37:11.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:11.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[21833]: from='osd.4 [v2:172.21.15.188:6800/1586199781,v1:172.21.15.188:6801/1586199781]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:11.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[21833]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:11.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[26549]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T21:37:11.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[26549]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T21:37:11.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:11.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[26549]: from='osd.4 [v2:172.21.15.188:6800/1586199781,v1:172.21.15.188:6801/1586199781]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:11.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:10 smithi165 ceph-mon[26549]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:11.170 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:10 smithi188 ceph-mon[22698]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T21:37:11.170 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:10 smithi188 ceph-mon[22698]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T21:37:11.170 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:10 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:11.170 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:10 smithi188 ceph-mon[22698]: from='osd.4 [v2:172.21.15.188:6800/1586199781,v1:172.21.15.188:6801/1586199781]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:11.170 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:10 smithi188 ceph-mon[22698]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:11.170 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:37:10 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:37:10.785+0000 7f3eacdf8700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T21:37:11.170 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:37:10 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:37:10.789+0000 7f3ea778f700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:37:12.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[21833]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:12.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[21833]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:37:12.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[21833]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T21:37:12.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:12.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:12.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[26549]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:12.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[26549]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:37:12.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[26549]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T21:37:12.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:12.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:11 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:12.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:11 smithi188 ceph-mon[22698]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:12.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:11 smithi188 ceph-mon[22698]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:37:12.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:11 smithi188 ceph-mon[22698]: osdmap e30: 5 total, 4 up, 5 in 2024-04-22T21:37:12.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:12.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:11 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:13.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:12 smithi188 ceph-mon[22698]: purged_snaps scrub starts 2024-04-22T21:37:13.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:12 smithi188 ceph-mon[22698]: purged_snaps scrub ok 2024-04-22T21:37:13.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:12 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:13.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:12 smithi188 ceph-mon[22698]: osd.4 [v2:172.21.15.188:6800/1586199781,v1:172.21.15.188:6801/1586199781] boot 2024-04-22T21:37:13.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:12 smithi188 ceph-mon[22698]: osdmap e31: 5 total, 5 up, 5 in 2024-04-22T21:37:13.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:12 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:13.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:12 smithi188 ceph-mon[22698]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T21:37:13.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[21833]: purged_snaps scrub starts 2024-04-22T21:37:13.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[21833]: purged_snaps scrub ok 2024-04-22T21:37:13.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:13.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[21833]: osd.4 [v2:172.21.15.188:6800/1586199781,v1:172.21.15.188:6801/1586199781] boot 2024-04-22T21:37:13.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[21833]: osdmap e31: 5 total, 5 up, 5 in 2024-04-22T21:37:13.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:13.059 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[21833]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T21:37:13.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[26549]: purged_snaps scrub starts 2024-04-22T21:37:13.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[26549]: purged_snaps scrub ok 2024-04-22T21:37:13.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:13.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[26549]: osd.4 [v2:172.21.15.188:6800/1586199781,v1:172.21.15.188:6801/1586199781] boot 2024-04-22T21:37:13.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[26549]: osdmap e31: 5 total, 5 up, 5 in 2024-04-22T21:37:13.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:37:13.060 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:12 smithi165 ceph-mon[26549]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T21:37:13.324 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T21:37:13.324 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T21:37:13.324 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-22T21:37:13.324 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-22T21:37:13.324 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133601 s, 785 MB/s 2024-04-22T21:37:13.324 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-22T21:37:14.037 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:13 smithi188 ceph-mon[22698]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:14.037 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:13 smithi188 ceph-mon[22698]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T21:37:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:13 smithi165 ceph-mon[21833]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:13 smithi165 ceph-mon[21833]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T21:37:14.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:13 smithi165 ceph-mon[26549]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T21:37:14.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:13 smithi165 ceph-mon[26549]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T21:37:14.063 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2024-04-22T21:37:15.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:15 smithi188 ceph-mon[22698]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:15.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:15 smithi188 ceph-mon[22698]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T21:37:15.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:15 smithi188 ceph-mon[22698]: Detected new or changed devices on smithi188 2024-04-22T21:37:15.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:15 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:15.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:15 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:37:15.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:15 smithi188 ceph-mon[22698]: Adjusting osd_memory_target on smithi188 to 17088M 2024-04-22T21:37:15.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:15 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:15.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:15 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:15.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[26549]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:15.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[26549]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T21:37:15.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[26549]: Detected new or changed devices on smithi188 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[26549]: Adjusting osd_memory_target on smithi188 to 17088M 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[21833]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[21833]: osdmap e34: 5 total, 5 up, 5 in 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[21833]: Detected new or changed devices on smithi188 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:15.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:37:15.810 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[21833]: Adjusting osd_memory_target on smithi188 to 17088M 2024-04-22T21:37:15.810 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:15.810 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:15 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:16.458 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:16 smithi188 ceph-mon[22698]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T21:37:16.459 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:37:16.459 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:37:16.459 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:16.459 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:37:16.459 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:37:16.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[21833]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T21:37:16.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:37:16.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:37:16.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:16.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:37:16.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:37:16.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[26549]: osdmap e35: 5 total, 5 up, 5 in 2024-04-22T21:37:16.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:37:16.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:37:16.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:16.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:37:16.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:37:17.455 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:17 smithi188 ceph-mon[22698]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:37:17.456 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:17 smithi188 ceph-mon[22698]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:17.751 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:17 smithi165 ceph-mon[21833]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:37:17.751 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:17 smithi165 ceph-mon[21833]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:17.751 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:17 smithi165 ceph-mon[26549]: from='client.34226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:37:17.751 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:17 smithi165 ceph-mon[26549]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:19.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:19 smithi188 ceph-mon[22698]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-22T21:37:19.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:19 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/2386874332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecfecc-9483-4c4c-98ce-25a89006dcd5"}]: dispatch 2024-04-22T21:37:19.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:19 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecfecc-9483-4c4c-98ce-25a89006dcd5"}]: dispatch 2024-04-22T21:37:19.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:19 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75ecfecc-9483-4c4c-98ce-25a89006dcd5"}]': finished 2024-04-22T21:37:19.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:19 smithi188 ceph-mon[22698]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T21:37:19.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:19 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:19.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:19 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/962492472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:37:19.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[26549]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-22T21:37:19.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/2386874332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecfecc-9483-4c4c-98ce-25a89006dcd5"}]: dispatch 2024-04-22T21:37:19.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecfecc-9483-4c4c-98ce-25a89006dcd5"}]: dispatch 2024-04-22T21:37:19.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75ecfecc-9483-4c4c-98ce-25a89006dcd5"}]': finished 2024-04-22T21:37:19.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[26549]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T21:37:19.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:19.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/962492472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:37:19.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[21833]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-22T21:37:19.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/2386874332' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecfecc-9483-4c4c-98ce-25a89006dcd5"}]: dispatch 2024-04-22T21:37:19.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75ecfecc-9483-4c4c-98ce-25a89006dcd5"}]: dispatch 2024-04-22T21:37:19.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75ecfecc-9483-4c4c-98ce-25a89006dcd5"}]': finished 2024-04-22T21:37:19.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[21833]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T21:37:19.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:19.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:19 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/962492472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:37:21.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:21 smithi188 ceph-mon[22698]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:37:21.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:21 smithi165 ceph-mon[21833]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:37:21.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:21 smithi165 ceph-mon[26549]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:37:23.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:23 smithi188 ceph-mon[22698]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-22T21:37:23.610 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:23 smithi165 ceph-mon[26549]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-22T21:37:23.611 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:23 smithi165 ceph-mon[21833]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-22T21:37:25.703 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:25 smithi188 ceph-mon[22698]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T21:37:25.704 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:25 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T21:37:25.704 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:25 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:25.704 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:25 smithi188 ceph-mon[22698]: Deploying daemon osd.5 on smithi188 2024-04-22T21:37:25.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:25 smithi165 ceph-mon[21833]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T21:37:25.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:25 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T21:37:25.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:25 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:25.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:25 smithi165 ceph-mon[21833]: Deploying daemon osd.5 on smithi188 2024-04-22T21:37:25.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:25 smithi165 ceph-mon[26549]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-22T21:37:25.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:25 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T21:37:25.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:25 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:25.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:25 smithi165 ceph-mon[26549]: Deploying daemon osd.5 on smithi188 2024-04-22T21:37:27.662 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:27 smithi188 ceph-mon[22698]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T21:37:27.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:27 smithi165 ceph-mon[26549]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T21:37:27.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:27 smithi165 ceph-mon[21833]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-22T21:37:29.706 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:29 smithi188 ceph-mon[22698]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:29.706 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:29 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:29.706 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:29 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:29.706 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:29 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:29.707 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:29 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:29.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[21833]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:29.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:29.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:29.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:29.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:29.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[26549]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:29.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:29.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:29.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:29.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:29 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:31.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:31 smithi188 ceph-mon[22698]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:31.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:31 smithi165 ceph-mon[21833]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:31.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:31 smithi165 ceph-mon[26549]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:32.235 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 5 on host 'smithi188' 2024-04-22T21:37:32.829 DEBUG:teuthology.orchestra.run.smithi188:osd.5> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T21:37:32.832 INFO:tasks.cephadm:Deploying osd.6 on smithi188 with /dev/vg_nvme/lv_2... 2024-04-22T21:37:32.832 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T21:37:33.487 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:33 smithi188 ceph-mon[22698]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:33.487 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:33 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:33.487 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:33 smithi188 ceph-mon[22698]: from='osd.5 [v2:172.21.15.188:6808/1947262752,v1:172.21.15.188:6809/1947262752]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:37:33.487 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:33 smithi188 ceph-mon[22698]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:37:33.487 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:33 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:33.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[21833]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:33.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:33.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[21833]: from='osd.5 [v2:172.21.15.188:6808/1947262752,v1:172.21.15.188:6809/1947262752]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:37:33.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[21833]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:37:33.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:33.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[26549]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:33.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:33.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[26549]: from='osd.5 [v2:172.21.15.188:6808/1947262752,v1:172.21.15.188:6809/1947262752]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:37:33.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[26549]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T21:37:33.559 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:33 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:34.488 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:34 smithi188 ceph-mon[22698]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T21:37:34.488 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:34 smithi188 ceph-mon[22698]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T21:37:34.488 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:34 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:34.488 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:34 smithi188 ceph-mon[22698]: from='osd.5 [v2:172.21.15.188:6808/1947262752,v1:172.21.15.188:6809/1947262752]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:34.488 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:34 smithi188 ceph-mon[22698]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:34.488 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:37:34 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:37:34.397+0000 7f1314ef9700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T21:37:34.488 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:37:34 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:37:34.402+0000 7f1310091700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:37:34.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[21833]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T21:37:34.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[21833]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T21:37:34.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:34.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[21833]: from='osd.5 [v2:172.21.15.188:6808/1947262752,v1:172.21.15.188:6809/1947262752]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:34.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[21833]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:34.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[26549]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T21:37:34.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[26549]: osdmap e37: 6 total, 5 up, 6 in 2024-04-22T21:37:34.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:34.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[26549]: from='osd.5 [v2:172.21.15.188:6808/1947262752,v1:172.21.15.188:6809/1947262752]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:34.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:34 smithi165 ceph-mon[26549]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:35.419 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T21:37:35.419 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T21:37:35.419 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-22T21:37:35.419 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-22T21:37:35.419 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0183938 s, 570 MB/s 2024-04-22T21:37:35.419 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-22T21:37:35.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:35 smithi188 ceph-mon[22698]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:35.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:35 smithi188 ceph-mon[22698]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:37:35.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:35 smithi188 ceph-mon[22698]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T21:37:35.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:35 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:35.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:35 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:35.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[21833]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:35.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[21833]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:37:35.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[21833]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T21:37:35.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:35.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:35.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[26549]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T21:37:35.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[26549]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:37:35.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[26549]: osdmap e38: 6 total, 5 up, 6 in 2024-04-22T21:37:35.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:35.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:35 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:36.054 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2024-04-22T21:37:36.593 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:36 smithi188 ceph-mon[22698]: purged_snaps scrub starts 2024-04-22T21:37:36.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:36 smithi188 ceph-mon[22698]: purged_snaps scrub ok 2024-04-22T21:37:36.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:36 smithi188 ceph-mon[22698]: osd.5 [v2:172.21.15.188:6808/1947262752,v1:172.21.15.188:6809/1947262752] boot 2024-04-22T21:37:36.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:36 smithi188 ceph-mon[22698]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T21:37:36.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:36 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:36.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:36 smithi188 ceph-mon[22698]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T21:37:36.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[21833]: purged_snaps scrub starts 2024-04-22T21:37:36.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[21833]: purged_snaps scrub ok 2024-04-22T21:37:36.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[21833]: osd.5 [v2:172.21.15.188:6808/1947262752,v1:172.21.15.188:6809/1947262752] boot 2024-04-22T21:37:36.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[21833]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T21:37:36.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:36.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[21833]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T21:37:36.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[26549]: purged_snaps scrub starts 2024-04-22T21:37:36.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[26549]: purged_snaps scrub ok 2024-04-22T21:37:36.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[26549]: osd.5 [v2:172.21.15.188:6808/1947262752,v1:172.21.15.188:6809/1947262752] boot 2024-04-22T21:37:36.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[26549]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T21:37:36.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:37:36.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:36 smithi165 ceph-mon[26549]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T21:37:37.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:37 smithi188 ceph-mon[22698]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:37.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:37 smithi188 ceph-mon[22698]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T21:37:37.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:37 smithi165 ceph-mon[21833]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:37.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:37 smithi165 ceph-mon[21833]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T21:37:37.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:37 smithi165 ceph-mon[26549]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:37.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:37 smithi165 ceph-mon[26549]: osdmap e41: 6 total, 6 up, 6 in 2024-04-22T21:37:39.133 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: Detected new or changed devices on smithi188 2024-04-22T21:37:39.133 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.133 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:37:39.134 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:37:39.134 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: Adjusting osd_memory_target on smithi188 to 8544M 2024-04-22T21:37:39.134 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.134 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.134 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:39.134 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:39.134 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:39.134 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.134 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:38 smithi188 ceph-mon[22698]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T21:37:39.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: Detected new or changed devices on smithi188 2024-04-22T21:37:39.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:37:39.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:37:39.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: Adjusting osd_memory_target on smithi188 to 8544M 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[21833]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: Detected new or changed devices on smithi188 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: Adjusting osd_memory_target on smithi188 to 8544M 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:39.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:39.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:39.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:39.310 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:38 smithi165 ceph-mon[26549]: osdmap e42: 6 total, 6 up, 6 in 2024-04-22T21:37:40.199 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:39 smithi188 ceph-mon[22698]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:40.199 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:39 smithi188 ceph-mon[22698]: from='client.34253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:37:40.199 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:39 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:37:40.199 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:39 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:37:40.199 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:39 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:40.199 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:39 smithi188 ceph-mon[22698]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T21:37:40.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[21833]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:40.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[21833]: from='client.34253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:37:40.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:37:40.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:37:40.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:40.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[21833]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T21:37:40.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[26549]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:40.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[26549]: from='client.34253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:37:40.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:37:40.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:37:40.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:40.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:39 smithi165 ceph-mon[26549]: osdmap e43: 6 total, 6 up, 6 in 2024-04-22T21:37:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:41 smithi188 ceph-mon[22698]: pgmap v135: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:41 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/818249800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eeceb5bb-7317-422b-9d66-62e099e57b49"}]: dispatch 2024-04-22T21:37:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:41 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eeceb5bb-7317-422b-9d66-62e099e57b49"}]: dispatch 2024-04-22T21:37:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:41 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eeceb5bb-7317-422b-9d66-62e099e57b49"}]': finished 2024-04-22T21:37:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:41 smithi188 ceph-mon[22698]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T21:37:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:41 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:42.258 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:41 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/1835574892' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:37:42.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[21833]: pgmap v135: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:42.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/818249800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eeceb5bb-7317-422b-9d66-62e099e57b49"}]: dispatch 2024-04-22T21:37:42.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eeceb5bb-7317-422b-9d66-62e099e57b49"}]: dispatch 2024-04-22T21:37:42.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eeceb5bb-7317-422b-9d66-62e099e57b49"}]': finished 2024-04-22T21:37:42.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[21833]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T21:37:42.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:42.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/1835574892' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:37:42.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[26549]: pgmap v135: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:42.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/818249800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eeceb5bb-7317-422b-9d66-62e099e57b49"}]: dispatch 2024-04-22T21:37:42.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eeceb5bb-7317-422b-9d66-62e099e57b49"}]: dispatch 2024-04-22T21:37:42.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eeceb5bb-7317-422b-9d66-62e099e57b49"}]': finished 2024-04-22T21:37:42.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[26549]: osdmap e44: 7 total, 6 up, 7 in 2024-04-22T21:37:42.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:42.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:41 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/1835574892' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:37:44.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:43 smithi188 ceph-mon[22698]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:44.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:43 smithi165 ceph-mon[21833]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:44.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:43 smithi165 ceph-mon[26549]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:45.967 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:45 smithi188 ceph-mon[22698]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:46.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:45 smithi165 ceph-mon[21833]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:46.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:45 smithi165 ceph-mon[26549]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:47 smithi188 ceph-mon[22698]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:47 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T21:37:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:47 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:48.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:47 smithi165 ceph-mon[21833]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:48.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:47 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T21:37:48.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:47 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:48.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:47 smithi165 ceph-mon[26549]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:48.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T21:37:48.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:47 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:48.976 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:48 smithi188 ceph-mon[22698]: Deploying daemon osd.6 on smithi188 2024-04-22T21:37:49.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:48 smithi165 ceph-mon[21833]: Deploying daemon osd.6 on smithi188 2024-04-22T21:37:49.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:48 smithi165 ceph-mon[26549]: Deploying daemon osd.6 on smithi188 2024-04-22T21:37:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:49 smithi188 ceph-mon[22698]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:50.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:49 smithi165 ceph-mon[21833]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:50.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:49 smithi165 ceph-mon[26549]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:52.137 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:51 smithi188 ceph-mon[22698]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:52.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:51 smithi165 ceph-mon[21833]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:52.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:51 smithi165 ceph-mon[26549]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:53.754 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:53 smithi188 ceph-mon[22698]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:53.754 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:53 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:53.754 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:53 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:53.754 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:53 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:53.754 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:53 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:53.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[21833]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:53.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:53.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:53.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:53.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:53.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[26549]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:53.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:53.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:37:53.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:37:53.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:53 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:37:55.490 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:55 smithi188 ceph-mon[22698]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:55.546 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 6 on host 'smithi188' 2024-04-22T21:37:55.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:55 smithi165 ceph-mon[21833]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:55.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:55 smithi165 ceph-mon[26549]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:56.240 DEBUG:teuthology.orchestra.run.smithi188:osd.6> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.6.service 2024-04-22T21:37:56.243 INFO:tasks.cephadm:Deploying osd.7 on smithi188 with /dev/vg_nvme/lv_1... 2024-04-22T21:37:56.243 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T21:37:56.474 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:56 smithi188 ceph-mon[22698]: from='osd.6 [v2:172.21.15.188:6816/1697563430,v1:172.21.15.188:6817/1697563430]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:37:56.474 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:56 smithi188 ceph-mon[22698]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:37:56.474 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:56 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:56.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:56 smithi165 ceph-mon[21833]: from='osd.6 [v2:172.21.15.188:6816/1697563430,v1:172.21.15.188:6817/1697563430]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:37:56.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:56 smithi165 ceph-mon[21833]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:37:56.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:56 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:56.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:56 smithi165 ceph-mon[26549]: from='osd.6 [v2:172.21.15.188:6816/1697563430,v1:172.21.15.188:6817/1697563430]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:37:56.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:56 smithi165 ceph-mon[26549]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T21:37:56.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:56 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:57.757 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:37:57 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:37:57.465+0000 7fa5e8a80700 -1 osd.6 0 waiting for initial osdmap 2024-04-22T21:37:57.758 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:37:57 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:37:57.472+0000 7fa5e3c18700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:37:57.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:57 smithi188 ceph-mon[22698]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:57.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:57 smithi188 ceph-mon[22698]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T21:37:57.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:57 smithi188 ceph-mon[22698]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T21:37:57.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:57.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:57 smithi188 ceph-mon[22698]: from='osd.6 [v2:172.21.15.188:6816/1697563430,v1:172.21.15.188:6817/1697563430]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:57.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:57 smithi188 ceph-mon[22698]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:57.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:57 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:57.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[21833]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:57.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[21833]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T21:37:57.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[21833]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T21:37:57.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:57.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[21833]: from='osd.6 [v2:172.21.15.188:6816/1697563430,v1:172.21.15.188:6817/1697563430]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:57.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[21833]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:57.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:57.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[26549]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:57.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[26549]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T21:37:57.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[26549]: osdmap e45: 7 total, 6 up, 7 in 2024-04-22T21:37:57.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:57.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[26549]: from='osd.6 [v2:172.21.15.188:6816/1697563430,v1:172.21.15.188:6817/1697563430]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:57.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[26549]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:37:57.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:57 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:37:58.262 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T21:37:58.263 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T21:37:58.263 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-22T21:37:58.263 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-22T21:37:58.263 INFO:teuthology.orchestra.run.smithi188.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0153742 s, 682 MB/s 2024-04-22T21:37:58.263 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-22T21:37:58.715 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:58 smithi188 ceph-mon[22698]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:37:58.715 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:58 smithi188 ceph-mon[22698]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T21:37:58.715 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:58 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:58.715 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:58 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:58.716 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:58 smithi188 ceph-mon[22698]: osd.6 [v2:172.21.15.188:6816/1697563430,v1:172.21.15.188:6817/1697563430] boot 2024-04-22T21:37:58.716 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:58 smithi188 ceph-mon[22698]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T21:37:58.716 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:58 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:58.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[21833]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:37:58.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[21833]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T21:37:58.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:58.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:58.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[21833]: osd.6 [v2:172.21.15.188:6816/1697563430,v1:172.21.15.188:6817/1697563430] boot 2024-04-22T21:37:58.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[21833]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T21:37:58.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:58.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[26549]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:37:58.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[26549]: osdmap e46: 7 total, 6 up, 7 in 2024-04-22T21:37:58.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:58.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:58.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[26549]: osd.6 [v2:172.21.15.188:6816/1697563430,v1:172.21.15.188:6817/1697563430] boot 2024-04-22T21:37:58.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[26549]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T21:37:58.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:58 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:37:59.622 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:59 smithi188 ceph-mon[22698]: purged_snaps scrub starts 2024-04-22T21:37:59.622 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:59 smithi188 ceph-mon[22698]: purged_snaps scrub ok 2024-04-22T21:37:59.622 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:59 smithi188 ceph-mon[22698]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:59.622 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:37:59 smithi188 ceph-mon[22698]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T21:37:59.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:59 smithi165 ceph-mon[21833]: purged_snaps scrub starts 2024-04-22T21:37:59.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:59 smithi165 ceph-mon[21833]: purged_snaps scrub ok 2024-04-22T21:37:59.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:59 smithi165 ceph-mon[21833]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:59.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:37:59 smithi165 ceph-mon[21833]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T21:37:59.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:59 smithi165 ceph-mon[26549]: purged_snaps scrub starts 2024-04-22T21:37:59.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:59 smithi165 ceph-mon[26549]: purged_snaps scrub ok 2024-04-22T21:37:59.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:59 smithi165 ceph-mon[26549]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T21:37:59.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:37:59 smithi165 ceph-mon[26549]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T21:38:00.192 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi188:vg_nvme/lv_1 2024-04-22T21:38:01.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:01.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: Detected new or changed devices on smithi188 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: Adjusting osd_memory_target on smithi188 to 5696M 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:01.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:01 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:01.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T21:38:01.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: Detected new or changed devices on smithi188 2024-04-22T21:38:01.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:01.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:01.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:01.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: Adjusting osd_memory_target on smithi188 to 5696M 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: osdmap e49: 7 total, 7 up, 7 in 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: Detected new or changed devices on smithi188 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:01.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:01.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: Adjusting osd_memory_target on smithi188 to 5696M 2024-04-22T21:38:01.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:01.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:01.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:01.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:01.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:01 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:02.705 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:02 smithi188 ceph-mon[22698]: osdmap e50: 7 total, 7 up, 7 in 2024-04-22T21:38:02.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:02 smithi165 ceph-mon[21833]: osdmap e50: 7 total, 7 up, 7 in 2024-04-22T21:38:02.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:02 smithi165 ceph-mon[26549]: osdmap e50: 7 total, 7 up, 7 in 2024-04-22T21:38:03.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:03 smithi188 ceph-mon[22698]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:03.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:03 smithi188 ceph-mon[22698]: osdmap e51: 7 total, 7 up, 7 in 2024-04-22T21:38:03.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:03 smithi188 ceph-mon[22698]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:03.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:03 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:38:03.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:03 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:38:03.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:03 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:03.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[21833]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:03.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[21833]: osdmap e51: 7 total, 7 up, 7 in 2024-04-22T21:38:03.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[21833]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:03.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:38:03.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:38:03.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:03.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[26549]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:03.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[26549]: osdmap e51: 7 total, 7 up, 7 in 2024-04-22T21:38:03.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[26549]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:03.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T21:38:03.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T21:38:03.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:03 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:05.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:05 smithi188 ceph-mon[22698]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-22T21:38:05.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:05 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/1821401537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38a0f129-d414-410a-8e8a-37baa9ef6319"}]: dispatch 2024-04-22T21:38:05.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:05 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38a0f129-d414-410a-8e8a-37baa9ef6319"}]: dispatch 2024-04-22T21:38:05.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:05 smithi188 ceph-mon[22698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38a0f129-d414-410a-8e8a-37baa9ef6319"}]': finished 2024-04-22T21:38:05.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:05 smithi188 ceph-mon[22698]: osdmap e52: 8 total, 7 up, 8 in 2024-04-22T21:38:05.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:05 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:05.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[21833]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-22T21:38:05.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/1821401537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38a0f129-d414-410a-8e8a-37baa9ef6319"}]: dispatch 2024-04-22T21:38:05.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38a0f129-d414-410a-8e8a-37baa9ef6319"}]: dispatch 2024-04-22T21:38:05.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[21833]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38a0f129-d414-410a-8e8a-37baa9ef6319"}]': finished 2024-04-22T21:38:05.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[21833]: osdmap e52: 8 total, 7 up, 8 in 2024-04-22T21:38:05.809 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:05.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[26549]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-22T21:38:05.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/1821401537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38a0f129-d414-410a-8e8a-37baa9ef6319"}]: dispatch 2024-04-22T21:38:05.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38a0f129-d414-410a-8e8a-37baa9ef6319"}]: dispatch 2024-04-22T21:38:05.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[26549]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38a0f129-d414-410a-8e8a-37baa9ef6319"}]': finished 2024-04-22T21:38:05.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[26549]: osdmap e52: 8 total, 7 up, 8 in 2024-04-22T21:38:05.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:05 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:06.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:06 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/508111998' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:38:06.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:06 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/508111998' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:38:06.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:06 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/508111998' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T21:38:07.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:07 smithi188 ceph-mon[22698]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T21:38:07.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:07 smithi165 ceph-mon[21833]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T21:38:07.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:07 smithi165 ceph-mon[26549]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-22T21:38:09.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:09 smithi188 ceph-mon[22698]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-22T21:38:09.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:09 smithi165 ceph-mon[21833]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-22T21:38:09.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:09 smithi165 ceph-mon[26549]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-22T21:38:11.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:11 smithi188 ceph-mon[22698]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-22T21:38:11.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:11 smithi165 ceph-mon[21833]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-22T21:38:11.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:11 smithi165 ceph-mon[26549]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-22T21:38:12.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:12 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T21:38:12.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:12 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:12.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:12 smithi188 ceph-mon[22698]: Deploying daemon osd.7 on smithi188 2024-04-22T21:38:12.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:12 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T21:38:12.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:12 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:12.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:12 smithi165 ceph-mon[26549]: Deploying daemon osd.7 on smithi188 2024-04-22T21:38:12.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:12 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T21:38:12.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:12 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:12.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:12 smithi165 ceph-mon[21833]: Deploying daemon osd.7 on smithi188 2024-04-22T21:38:13.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:13 smithi188 ceph-mon[22698]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-22T21:38:13.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:13 smithi165 ceph-mon[26549]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-22T21:38:13.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:13 smithi165 ceph-mon[21833]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-22T21:38:15.670 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:15 smithi188 ceph-mon[22698]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:15.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:15 smithi165 ceph-mon[21833]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:15.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:15 smithi165 ceph-mon[26549]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:16.588 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:38:16.589 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:16 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:38:16.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:38:16.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:16 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:38:16.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:38:16.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:16 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:38:17.956 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:17 smithi188 ceph-mon[22698]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:17.956 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:17 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:17.956 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:17 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:17.956 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:17 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:17.956 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:17 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:18.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[21833]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:18.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:18.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:18.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:18.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:18.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[26549]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:18.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:18.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:18.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:18.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:17 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:19.695 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:19 smithi188 ceph-mon[22698]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:19.765 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 7 on host 'smithi188' 2024-04-22T21:38:20.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:19 smithi165 ceph-mon[21833]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:20.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:19 smithi165 ceph-mon[26549]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:20.334 DEBUG:teuthology.orchestra.run.smithi188:osd.7> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.7.service 2024-04-22T21:38:20.336 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-22T21:38:20.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T21:38:20.651 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:20 smithi165 ceph-mon[26549]: from='osd.7 [v2:172.21.15.188:6824/164704082,v1:172.21.15.188:6825/164704082]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:38:20.651 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:20 smithi165 ceph-mon[26549]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:38:20.651 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:20 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:20.651 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:20 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:20.652 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:20 smithi165 ceph-mon[21833]: from='osd.7 [v2:172.21.15.188:6824/164704082,v1:172.21.15.188:6825/164704082]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:38:20.652 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:20 smithi165 ceph-mon[21833]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:38:20.652 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:20 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:20.652 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:20 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:20.903 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:20 smithi188 ceph-mon[22698]: from='osd.7 [v2:172.21.15.188:6824/164704082,v1:172.21.15.188:6825/164704082]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:38:20.904 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:20 smithi188 ceph-mon[22698]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T21:38:20.904 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:20 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:20.904 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:20 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:21.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:21 smithi188 ceph-mon[22698]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:21.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:21 smithi188 ceph-mon[22698]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T21:38:21.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:21 smithi188 ceph-mon[22698]: osdmap e53: 8 total, 7 up, 8 in 2024-04-22T21:38:21.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:21 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:21.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:21 smithi188 ceph-mon[22698]: from='osd.7 [v2:172.21.15.188:6824/164704082,v1:172.21.15.188:6825/164704082]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:38:21.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:21 smithi188 ceph-mon[22698]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:38:21.759 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:38:21 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:38:21.638+0000 7f3cd4178700 -1 osd.7 0 waiting for initial osdmap 2024-04-22T21:38:21.759 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:38:21 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:38:21.645+0000 7f3cccb0b700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T21:38:21.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[26549]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:21.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[26549]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T21:38:21.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[26549]: osdmap e53: 8 total, 7 up, 8 in 2024-04-22T21:38:21.888 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:21.888 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[26549]: from='osd.7 [v2:172.21.15.188:6824/164704082,v1:172.21.15.188:6825/164704082]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:38:21.888 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[26549]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:38:21.888 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[21833]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:21.888 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[21833]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T21:38:21.888 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[21833]: osdmap e53: 8 total, 7 up, 8 in 2024-04-22T21:38:21.888 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:21.888 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[21833]: from='osd.7 [v2:172.21.15.188:6824/164704082,v1:172.21.15.188:6825/164704082]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:38:21.889 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:21 smithi165 ceph-mon[21833]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-22T21:38:22.624 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:38:22.993 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[21833]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:38:22.993 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[21833]: osdmap e54: 8 total, 7 up, 8 in 2024-04-22T21:38:22.993 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[21833]: osd.7 [v2:172.21.15.188:6824/164704082,v1:172.21.15.188:6825/164704082] boot 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[21833]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[26549]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[26549]: osdmap e54: 8 total, 7 up, 8 in 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[26549]: osd.7 [v2:172.21.15.188:6824/164704082,v1:172.21.15.188:6825/164704082] boot 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[26549]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T21:38:22.994 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:22 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:23.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:22 smithi188 ceph-mon[22698]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-22T21:38:23.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:22 smithi188 ceph-mon[22698]: osdmap e54: 8 total, 7 up, 8 in 2024-04-22T21:38:23.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:22 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:23.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:22 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:23.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:22 smithi188 ceph-mon[22698]: osd.7 [v2:172.21.15.188:6824/164704082,v1:172.21.15.188:6825/164704082] boot 2024-04-22T21:38:23.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:22 smithi188 ceph-mon[22698]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T21:38:23.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:22 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:23.460 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1713821902,"num_in_osds":8,"osd_in_since":1713821885,"num_remapped_pgs":1} 2024-04-22T21:38:23.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T21:38:23.678 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:38:23.716 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: purged_snaps scrub starts 2024-04-22T21:38:23.716 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: purged_snaps scrub ok 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1961910671' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:23.717 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: purged_snaps scrub starts 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: purged_snaps scrub ok 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1961910671' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.718 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.719 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.719 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:23.719 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:23.719 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:23.719 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:23 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: purged_snaps scrub starts 2024-04-22T21:38:23.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: purged_snaps scrub ok 2024-04-22T21:38:23.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T21:38:23.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1961910671' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:23.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:23 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:24.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:24 smithi165 ceph-mon[26549]: Detected new or changed devices on smithi188 2024-04-22T21:38:24.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:24 smithi165 ceph-mon[26549]: Adjusting osd_memory_target on smithi188 to 4272M 2024-04-22T21:38:24.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:24 smithi165 ceph-mon[26549]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T21:38:24.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:24 smithi165 ceph-mon[21833]: Detected new or changed devices on smithi188 2024-04-22T21:38:24.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:24 smithi165 ceph-mon[21833]: Adjusting osd_memory_target on smithi188 to 4272M 2024-04-22T21:38:24.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:24 smithi165 ceph-mon[21833]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T21:38:25.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:24 smithi188 ceph-mon[22698]: Detected new or changed devices on smithi188 2024-04-22T21:38:25.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:24 smithi188 ceph-mon[22698]: Adjusting osd_memory_target on smithi188 to 4272M 2024-04-22T21:38:25.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:24 smithi188 ceph-mon[22698]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T21:38:25.597 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:38:25.598 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":58,"fsid":"dccb09a4-00ef-11ef-bc93-c7b262605968","created":"2024-04-22T21:33:22.831640+0000","modified":"2024-04-22T21:38:25.363776+0000","last_up_change":"2024-04-22T21:38:22.352005+0000","last_in_change":"2024-04-22T21:38:05.201912+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:36:26.472879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"315d5f25-fb61-4586-b222-e4463c47ac34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6803","nonce":3819369935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6805","nonce":3819369935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6809","nonce":3819369935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6807","nonce":3819369935}]},"public_addr":"172.21.15.165:6803/3819369935","cluster_addr":"172.21.15.165:6805/3819369935","heartbeat_back_addr":"172.21.15.165:6809/3819369935","heartbeat_front_addr":"172.21.15.165:6807/3819369935","state":["exists","up"]},{"osd":1,"uuid":"8e61383e-0138-4fb4-ac5a-1a9f96406576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6811","nonce":4196345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6813","nonce":4196345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6817","nonce":4196345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6815","nonce":4196345941}]},"public_addr":"172.21.15.165:6811/4196345941","cluster_addr":"172.21.15.165:6813/4196345941","heartbeat_back_addr":"172.21.15.165:6817/4196345941","heartbeat_front_addr":"172.21.15.165:6815/4196345941","state":["exists","up"]},{"osd":2,"uuid":"0b1b573c-17c0-4e85-9f32-e66637e8fe0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6819","nonce":4051532826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6821","nonce":4051532826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6825","nonce":4051532826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6823","nonce":4051532826}]},"public_addr":"172.21.15.165:6819/4051532826","cluster_addr":"172.21.15.165:6821/4051532826","heartbeat_back_addr":"172.21.15.165:6825/4051532826","heartbeat_front_addr":"172.21.15.165:6823/4051532826","state":["exists","up"]},{"osd":3,"uuid":"5a6e8740-aefe-4aa7-a6bc-45a8fe626b24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6827","nonce":1668061591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6829","nonce":1668061591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6833","nonce":1668061591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6831","nonce":1668061591}]},"public_addr":"172.21.15.165:6827/1668061591","cluster_addr":"172.21.15.165:6829/1668061591","heartbeat_back_addr":"172.21.15.165:6833/1668061591","heartbeat_front_addr":"172.21.15.165:6831/1668061591","state":["exists","up"]},{"osd":4,"uuid":"dc54e2dc-af18-42ba-897c-b0e06fafb520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6801","nonce":1586199781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6803","nonce":1586199781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6807","nonce":1586199781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6805","nonce":1586199781}]},"public_addr":"172.21.15.188:6801/1586199781","cluster_addr":"172.21.15.188:6803/1586199781","heartbeat_back_addr":"172.21.15.188:6807/1586199781","heartbeat_front_addr":"172.21.15.188:6805/1586199781","state":["exists","up"]},{"osd":5,"uuid":"75ecfecc-9483-4c4c-98ce-25a89006dcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6809","nonce":1947262752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6811","nonce":1947262752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6815","nonce":1947262752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6813","nonce":1947262752}]},"public_addr":"172.21.15.188:6809/1947262752","cluster_addr":"172.21.15.188:6811/1947262752","heartbeat_back_addr":"172.21.15.188:6815/1947262752","heartbeat_front_addr":"172.21.15.188:6813/1947262752","state":["exists","up"]},{"osd":6,"uuid":"eeceb5bb-7317-422b-9d66-62e099e57b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6817","nonce":1697563430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6819","nonce":1697563430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6823","nonce":1697563430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6821","nonce":1697563430}]},"public_addr":"172.21.15.188:6817/1697563430","cluster_addr":"172.21.15.188:6819/1697563430","heartbeat_back_addr":"172.21.15.188:6823/1697563430","heartbeat_front_addr":"172.21.15.188:6821/1697563430","state":["exists","up"]},{"osd":7,"uuid":"38a0f129-d414-410a-8e8a-37baa9ef6319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6825","nonce":164704082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6827","nonce":164704082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6831","nonce":164704082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6829","nonce":164704082}]},"public_addr":"172.21.15.188:6825/164704082","cluster_addr":"172.21.15.188:6827/164704082","heartbeat_back_addr":"172.21.15.188:6831/164704082","heartbeat_front_addr":"172.21.15.188:6829/164704082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:35:34.798019+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:35:58.341829+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:36:22.899668+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:36:48.166539+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:37:10.294264+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:37:33.510025+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:37:56.516100+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:38:20.714448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:0/229165957":"2024-04-23T21:34:16.115619+0000","172.21.15.165:6801/2127003174":"2024-04-23T21:34:16.115619+0000","172.21.15.165:6800/2127003174":"2024-04-23T21:34:16.115619+0000","172.21.15.165:0/2427714119":"2024-04-23T21:33:52.472131+0000","172.21.15.165:6800/4281613920":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/1312953247":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/1483522820":"2024-04-23T21:34:16.115619+0000","172.21.15.165:6801/4281613920":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/3912587609":"2024-04-23T21:34:16.115619+0000","172.21.15.165:0/3416385233":"2024-04-23T21:33:52.472131+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:38:26.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:25 smithi188 ceph-mon[22698]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:26.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:25 smithi188 ceph-mon[22698]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T21:38:26.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:25 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/4224965454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:38:26.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:25 smithi165 ceph-mon[26549]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:26.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:25 smithi165 ceph-mon[26549]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T21:38:26.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:25 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/4224965454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:38:26.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:25 smithi165 ceph-mon[21833]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:26.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:25 smithi165 ceph-mon[21833]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T21:38:26.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:25 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/4224965454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:38:26.378 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T21:36:26.472879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-22T21:38:26.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T21:38:26.591 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:38:27.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:27 smithi188 ceph-mon[22698]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:27.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:27 smithi188 ceph-mon[22698]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T21:38:27.783 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:27 smithi165 ceph-mon[21833]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:27.783 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:27 smithi165 ceph-mon[21833]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T21:38:27.783 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:27 smithi165 ceph-mon[26549]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:27.783 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:27 smithi165 ceph-mon[26549]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T21:38:28.487 INFO:teuthology.orchestra.run.smithi165.stdout:pg_num: 1 2024-04-22T21:38:29.086 INFO:tasks.cephadm:Adding prometheus.a on smithi188 2024-04-22T21:38:29.087 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch apply prometheus '1;smithi188=a' 2024-04-22T21:38:29.400 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:29 smithi188 ceph-mon[22698]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-22T21:38:29.400 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:29 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1873349157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T21:38:29.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:29 smithi165 ceph-mon[26549]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-22T21:38:29.817 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:29 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1873349157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T21:38:29.818 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:29 smithi165 ceph-mon[21833]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-22T21:38:29.818 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:29 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1873349157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T21:38:31.019 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled prometheus update... 2024-04-22T21:38:31.620 DEBUG:teuthology.orchestra.run.smithi188:prometheus.a> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@prometheus.a.service 2024-04-22T21:38:31.623 INFO:tasks.cephadm:Adding node-exporter.a on smithi165 2024-04-22T21:38:31.623 INFO:tasks.cephadm:Adding node-exporter.b on smithi188 2024-04-22T21:38:31.623 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch apply node-exporter '2;smithi165=a;smithi188=b' 2024-04-22T21:38:32.232 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ignoring --setuser ceph since I am not root 2024-04-22T21:38:32.232 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ignoring --setgroup ceph since I am not root 2024-04-22T21:38:32.232 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mgr[23775]: -- 172.21.15.188:0/499837259 <== mon.2 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562ebd4a4340 con 0x562ebd47c800 2024-04-22T21:38:32.232 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:32.250+0000 7fe36300d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T21:38:32.232 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mon[22698]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:38:32.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mon[22698]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi188=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:32.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mon[22698]: Saving service prometheus spec with placement smithi188=a;count:1 2024-04-22T21:38:32.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:32.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:32.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:32.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:32.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:32.233 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T21:38:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[26549]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:38:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[26549]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi188=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[26549]: Saving service prometheus spec with placement smithi188=a;count:1 2024-04-22T21:38:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[21833]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[21833]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi188=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[21833]: Saving service prometheus spec with placement smithi188=a;count:1 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ignoring --setuser ceph since I am not root 2024-04-22T21:38:32.309 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ignoring --setgroup ceph since I am not root 2024-04-22T21:38:32.310 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-mgr[22067]: -- 172.21.15.165:0/1772347712 <== mon.1 v2:172.21.15.165:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fe8d750340 con 0x55fe8d728c00 2024-04-22T21:38:32.310 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:32.250+0000 7f5beb083000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T21:38:32.507 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:32.380+0000 7fe36300d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:38:32.507 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:32.478+0000 7fe36300d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T21:38:32.759 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:32.377+0000 7f5beb083000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:38:32.759 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:32.474+0000 7f5beb083000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T21:38:32.759 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:32.776+0000 7f5beb083000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T21:38:32.971 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:32.791+0000 7fe36300d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T21:38:32.971 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:32 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:32.873+0000 7fe36300d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T21:38:33.057 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:32 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:32.858+0000 7f5beb083000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T21:38:33.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:33 smithi188 ceph-mon[22698]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T21:38:33.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:33 smithi188 ceph-mon[22698]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-22T21:38:33.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:33 smithi165 ceph-mon[21833]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T21:38:33.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:33 smithi165 ceph-mon[21833]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-22T21:38:33.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:33 smithi165 ceph-mon[26549]: from='mgr.14150 172.21.15.165:0/2539813818' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T21:38:33.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:33 smithi165 ceph-mon[26549]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-22T21:38:34.257 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:34 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:34.020+0000 7fe36300d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T21:38:34.257 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:34 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:34.184+0000 7fe36300d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T21:38:34.258 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:34 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:34.266+0000 7fe36300d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T21:38:34.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:33.967+0000 7f5beb083000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T21:38:34.308 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:34.131+0000 7f5beb083000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T21:38:34.308 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:34.216+0000 7f5beb083000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T21:38:34.808 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:34.527+0000 7f5beb083000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T21:38:34.808 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:34.617+0000 7f5beb083000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T21:38:34.808 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:34.713+0000 7f5beb083000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T21:38:34.983 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:34 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:34.578+0000 7fe36300d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T21:38:34.983 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:34 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:34.668+0000 7fe36300d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T21:38:34.983 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:34 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:34.759+0000 7fe36300d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T21:38:34.983 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:35 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:35.001+0000 7fe36300d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:38:35.308 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:34 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:34.958+0000 7f5beb083000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T21:38:35.757 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:35 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:35.437+0000 7fe36300d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T21:38:35.807 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:35 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:35.394+0000 7f5beb083000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T21:38:36.247 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:35 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:35.879+0000 7f5beb083000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T21:38:36.247 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:35 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:35.972+0000 7f5beb083000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T21:38:36.257 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:35 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:35.921+0000 7fe36300d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T21:38:36.257 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:36 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:36.014+0000 7fe36300d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T21:38:36.557 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:36 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:36.265+0000 7f5beb083000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T21:38:36.558 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:36 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:36.356+0000 7f5beb083000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T21:38:36.757 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:36 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:36.306+0000 7fe36300d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T21:38:36.757 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:36 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:36.394+0000 7fe36300d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T21:38:37.058 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:36 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:36.764+0000 7f5beb083000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T21:38:37.058 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:36 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:36.897+0000 7f5beb083000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T21:38:37.058 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:36 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:36.991+0000 7f5beb083000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T21:38:37.116 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:36 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:36.797+0000 7fe36300d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T21:38:37.116 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:36 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:36.930+0000 7fe36300d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T21:38:37.116 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:37 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:37.021+0000 7fe36300d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T21:38:37.116 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:37 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:37.135+0000 7fe36300d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T21:38:37.507 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:37 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:37.223+0000 7fe36300d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T21:38:37.558 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:37 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:37.108+0000 7f5beb083000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T21:38:37.558 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:37 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:37.198+0000 7f5beb083000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T21:38:38.007 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:37 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:37.768+0000 7fe36300d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T21:38:38.057 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:37 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:37.748+0000 7f5beb083000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T21:38:38.343 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:38.109+0000 7f5beb083000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T21:38:38.344 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[21833]: Active manager daemon y restarted 2024-04-22T21:38:38.344 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[21833]: Activating manager daemon y 2024-04-22T21:38:38.344 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[21833]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T21:38:38.344 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[21833]: Standby manager daemon x restarted 2024-04-22T21:38:38.344 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[21833]: Standby manager daemon x started 2024-04-22T21:38:38.344 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[21833]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T21:38:38.344 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[21833]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T21:38:38.345 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[21833]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T21:38:38.345 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[21833]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T21:38:38.436 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-mon[22698]: Active manager daemon y restarted 2024-04-22T21:38:38.436 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-mon[22698]: Activating manager daemon y 2024-04-22T21:38:38.436 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-mon[22698]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T21:38:38.436 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-mon[22698]: Standby manager daemon x restarted 2024-04-22T21:38:38.436 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-mon[22698]: Standby manager daemon x started 2024-04-22T21:38:38.436 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-mon[22698]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T21:38:38.437 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-mon[22698]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T21:38:38.437 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-mon[22698]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T21:38:38.437 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-mon[22698]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T21:38:38.437 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:38.130+0000 7fe36300d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T21:38:38.437 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: [22/Apr/2024:21:38:38] ENGINE Bus STARTING 2024-04-22T21:38:38.437 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: CherryPy Checker: 2024-04-22T21:38:38.437 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: The Application mounted at '' has an empty config. 2024-04-22T21:38:38.437 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: 2024-04-22T21:38:38.437 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: [22/Apr/2024:21:38:38] ENGINE Serving on http://:::9283 2024-04-22T21:38:38.438 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:38:38 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: [22/Apr/2024:21:38:38] ENGINE Bus STARTED 2024-04-22T21:38:38.646 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[26549]: Active manager daemon y restarted 2024-04-22T21:38:38.646 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[26549]: Activating manager daemon y 2024-04-22T21:38:38.646 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[26549]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T21:38:38.646 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[26549]: Standby manager daemon x restarted 2024-04-22T21:38:38.646 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[26549]: Standby manager daemon x started 2024-04-22T21:38:38.646 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[26549]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T21:38:38.646 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[26549]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T21:38:38.646 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[26549]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T21:38:38.646 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-mon[26549]: from='mgr.? 172.21.15.188:0/2955306763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T21:38:38.647 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:38:38] ENGINE Bus STARTING 2024-04-22T21:38:38.647 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: CherryPy Checker: 2024-04-22T21:38:38.647 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: The Application mounted at '' has an empty config. 2024-04-22T21:38:38.647 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:38:38.647 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:38:38] ENGINE Serving on http://:::9283 2024-04-22T21:38:38.647 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:38:38] ENGINE Bus STARTED 2024-04-22T21:38:38.962 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:38:38] ENGINE Bus STARTING 2024-04-22T21:38:38.962 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:38:38] ENGINE Serving on https://172.21.15.165:7150 2024-04-22T21:38:39.308 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:38:38 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: [22/Apr/2024:21:38:38] ENGINE Bus STARTED 2024-04-22T21:38:39.376 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled node-exporter update... 2024-04-22T21:38:39.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: mgrmap e17: y(active, starting, since 0.256733s), standbys: x 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:39.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: Manager daemon y is now available 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: [22/Apr/2024:21:38:38] ENGINE Bus STARTING 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: [22/Apr/2024:21:38:38] ENGINE Serving on https://172.21.15.165:7150 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: [22/Apr/2024:21:38:38] ENGINE Bus STARTED 2024-04-22T21:38:39.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: mgrmap e17: y(active, starting, since 0.256733s), standbys: x 2024-04-22T21:38:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:38:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:38:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:38:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T21:38:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T21:38:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:38:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: Manager daemon y is now available 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:39.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:38:39.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:38:39.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: [22/Apr/2024:21:38:38] ENGINE Bus STARTING 2024-04-22T21:38:39.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: [22/Apr/2024:21:38:38] ENGINE Serving on https://172.21.15.165:7150 2024-04-22T21:38:39.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: [22/Apr/2024:21:38:38] ENGINE Bus STARTED 2024-04-22T21:38:39.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:39.810 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: mgrmap e17: y(active, starting, since 0.256733s), standbys: x 2024-04-22T21:38:39.810 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T21:38:39.811 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T21:38:39.811 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T21:38:39.811 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T21:38:39.811 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T21:38:39.811 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T21:38:39.811 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T21:38:39.811 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T21:38:39.811 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T21:38:39.811 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: Manager daemon y is now available 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:38:39.812 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:38:39.813 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:39.813 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:38:39.813 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:38:39.813 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: [22/Apr/2024:21:38:38] ENGINE Bus STARTING 2024-04-22T21:38:39.813 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: [22/Apr/2024:21:38:38] ENGINE Serving on https://172.21.15.165:7150 2024-04-22T21:38:39.813 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: [22/Apr/2024:21:38:38] ENGINE Bus STARTED 2024-04-22T21:38:39.813 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:40.181 DEBUG:teuthology.orchestra.run.smithi165:node-exporter.a> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@node-exporter.a.service 2024-04-22T21:38:40.189 DEBUG:teuthology.orchestra.run.smithi188:node-exporter.b> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@node-exporter.b.service 2024-04-22T21:38:40.195 INFO:tasks.cephadm:Adding alertmanager.a on smithi165 2024-04-22T21:38:40.196 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch apply alertmanager '1;smithi165=a' 2024-04-22T21:38:40.470 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:40 smithi188 ceph-mon[22698]: mgrmap e18: y(active, since 1.27248s), standbys: x 2024-04-22T21:38:40.470 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:40 smithi188 ceph-mon[22698]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi165=a;smithi188=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:40.470 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:40 smithi188 ceph-mon[22698]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:40.471 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:40 smithi188 ceph-mon[22698]: Saving service node-exporter spec with placement smithi165=a;smithi188=b;count:2 2024-04-22T21:38:40.471 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:40 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:40.471 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:40 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:40.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[21833]: mgrmap e18: y(active, since 1.27248s), standbys: x 2024-04-22T21:38:40.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[21833]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi165=a;smithi188=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:40.524 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[21833]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:40.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[21833]: Saving service node-exporter spec with placement smithi165=a;smithi188=b;count:2 2024-04-22T21:38:40.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:40.525 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:40.525 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[26549]: mgrmap e18: y(active, since 1.27248s), standbys: x 2024-04-22T21:38:40.525 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[26549]: from='client.34334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi165=a;smithi188=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:40.525 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[26549]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:40.525 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[26549]: Saving service node-exporter spec with placement smithi165=a;smithi188=b;count:2 2024-04-22T21:38:40.525 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:40.525 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:40 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.593 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:41.593 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: Updating smithi165:/etc/ceph/ceph.conf 2024-04-22T21:38:41.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:38:41.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-22T21:38:41.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.594 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:41 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: Updating smithi165:/etc/ceph/ceph.conf 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: Updating smithi165:/etc/ceph/ceph.conf 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:41.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:41 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:42.308 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled alertmanager update... 2024-04-22T21:38:42.611 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:42 smithi188 ceph-mon[22698]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-22T21:38:42.611 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:42 smithi188 ceph-mon[22698]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:38:42.611 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:42 smithi188 ceph-mon[22698]: Deploying daemon prometheus.a on smithi188 2024-04-22T21:38:42.611 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:42 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:42.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:42 smithi165 ceph-mon[21833]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-22T21:38:42.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:42 smithi165 ceph-mon[21833]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:38:42.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:42 smithi165 ceph-mon[21833]: Deploying daemon prometheus.a on smithi188 2024-04-22T21:38:42.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:42 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:42.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:42 smithi165 ceph-mon[26549]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-22T21:38:42.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:42 smithi165 ceph-mon[26549]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-22T21:38:42.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:42 smithi165 ceph-mon[26549]: Deploying daemon prometheus.a on smithi188 2024-04-22T21:38:42.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:42 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:42.926 DEBUG:teuthology.orchestra.run.smithi165:alertmanager.a> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@alertmanager.a.service 2024-04-22T21:38:42.929 INFO:tasks.cephadm:Adding grafana.a on smithi188 2024-04-22T21:38:42.929 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph orch apply grafana '1;smithi188=a' 2024-04-22T21:38:43.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:43 smithi188 ceph-mon[22698]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:43.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:43 smithi188 ceph-mon[22698]: Saving service alertmanager spec with placement smithi165=a;count:1 2024-04-22T21:38:43.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:43 smithi188 ceph-mon[22698]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:43.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:43 smithi165 ceph-mon[21833]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:43.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:43 smithi165 ceph-mon[21833]: Saving service alertmanager spec with placement smithi165=a;count:1 2024-04-22T21:38:43.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:43 smithi165 ceph-mon[21833]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:43.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:43 smithi165 ceph-mon[26549]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:43.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:43 smithi165 ceph-mon[26549]: Saving service alertmanager spec with placement smithi165=a;count:1 2024-04-22T21:38:43.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:43 smithi165 ceph-mon[26549]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:45.642 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled grafana update... 2024-04-22T21:38:45.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:45 smithi188 ceph-mon[22698]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:45.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:45 smithi165 ceph-mon[21833]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:45.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:45 smithi165 ceph-mon[26549]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:46.675 DEBUG:teuthology.orchestra.run.smithi188:grafana.a> sudo journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@grafana.a.service 2024-04-22T21:38:46.677 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T21:38:46.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T21:38:46.987 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:46 smithi165 ceph-mon[21833]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi188=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:46.988 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:46 smithi165 ceph-mon[21833]: Saving service grafana spec with placement smithi188=a;count:1 2024-04-22T21:38:46.988 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:46 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:46.988 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:46 smithi165 ceph-mon[26549]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi188=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:46.988 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:46 smithi165 ceph-mon[26549]: Saving service grafana spec with placement smithi188=a;count:1 2024-04-22T21:38:46.988 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:46 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:47.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:46 smithi188 ceph-mon[22698]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi188=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:38:47.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:46 smithi188 ceph-mon[22698]: Saving service grafana spec with placement smithi188=a;count:1 2024-04-22T21:38:47.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:46 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:47.975 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:47 smithi165 ceph-mon[21833]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:47.976 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:47 smithi165 ceph-mon[26549]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:47 smithi188 ceph-mon[22698]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:48.688 INFO:teuthology.orchestra.run.smithi165.stdout:[client.0] 2024-04-22T21:38:48.689 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQDo2CZmajmzKRAAveC1yiQv23olJ1wM/j8V8A== 2024-04-22T21:38:49.363 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-04-22T21:38:49.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T21:38:49.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T21:38:49.399 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T21:38:49.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:49 smithi188 ceph-mon[22698]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:49.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:49 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/4045434471' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:49.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:49 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:49.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:49 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:38:50.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:49 smithi165 ceph-mon[21833]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:50.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:49 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/4045434471' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:50.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:49 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:50.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:49 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:38:50.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:49 smithi165 ceph-mon[26549]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:50.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:49 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/4045434471' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:50.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:49 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:50.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:49 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:38:51.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:51 smithi188 ceph-mon[22698]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:52.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:51 smithi165 ceph-mon[21833]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:52.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:51 smithi165 ceph-mon[26549]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:52.975 INFO:teuthology.orchestra.run.smithi188.stdout:[client.1] 2024-04-22T21:38:52.975 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQDs2CZmjFjtOhAAJFh3HaTZTdQdAS4ayfPkqg== 2024-04-22T21:38:53.607 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:53 smithi188 ceph-mon[22698]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:53.607 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:53 smithi188 ceph-mon[22698]: from='client.? 172.21.15.188:0/3529413801' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:53.607 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:53 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:53.608 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:53 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:38:53.631 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-22T21:38:53.631 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-22T21:38:53.631 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T21:38:53.668 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T21:38:53.668 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T21:38:53.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T21:38:53.694 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:53 smithi165 ceph-mon[21833]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:53.695 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:53 smithi165 ceph-mon[21833]: from='client.? 172.21.15.188:0/3529413801' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:53.695 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:53 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:53.695 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:53 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:38:53.695 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:53 smithi165 ceph-mon[26549]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:53.695 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:53 smithi165 ceph-mon[26549]: from='client.? 172.21.15.188:0/3529413801' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:53.695 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:53 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T21:38:53.695 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:53 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T21:38:53.896 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:38:55.654 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-mon[22698]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:55.654 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 systemd[1]: Starting Ceph prometheus.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:38:55.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:55 smithi165 ceph-mon[21833]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:55.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:55 smithi165 ceph-mon[26549]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:55.878 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 podman[37702]: 2024-04-22 21:38:55.57640462 +0000 UTC m=+0.018621750 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 podman[37702]: 2024-04-22 21:38:55.687324204 +0000 UTC m=+0.129541323 container create 70bc8fb45a33ef4058c19019076cad8567e9d35c2ef7b120fe68bce9765136e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 podman[37702]: 2024-04-22 21:38:55.897677723 +0000 UTC m=+0.339894846 container init 70bc8fb45a33ef4058c19019076cad8567e9d35c2ef7b120fe68bce9765136e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 podman[37702]: 2024-04-22 21:38:55.900338522 +0000 UTC m=+0.342555642 container start 70bc8fb45a33ef4058c19019076cad8567e9d35c2ef7b120fe68bce9765136e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.923Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.923Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.923Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.923Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi188 (none))" 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.923Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-22T21:38:55.905 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.923Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-22T21:38:56.257 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.924Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.925Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.925Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.928Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.928Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.766µs 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.928Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.928Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.928Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.966µs wal_replay_duration=133.038µs total_replay_duration=166.798µs 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.929Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.929Z caller=main.go:947 level=info msg="TSDB started" 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.929Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.944Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.528642ms db_storage=610ns remote_storage=1.136µs web_handler=253ns query_engine=476ns scrape=390.705µs scrape_sd=16.207µs notify=808ns notify_sd=1.387µs rules=14.95915ms 2024-04-22T21:38:56.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:38:55.944Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-22T21:38:56.259 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:56 smithi188 bash[37702]: 70bc8fb45a33ef4058c19019076cad8567e9d35c2ef7b120fe68bce9765136e7 2024-04-22T21:38:56.259 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:38:56 smithi188 systemd[1]: Started Ceph prometheus.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:38:56.636 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":19,"active_gid":24355,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":1360434262},{"type":"v1","addr":"172.21.15.165:6801","nonce":1360434262}]},"active_addr":"172.21.15.165:6801/1360434262","active_change":"2024-04-22T21:38:38.113149+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34322,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.165:8443/","prometheus":"http://172.21.15.165:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":3355385675}]},{"addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":2874487238}]},{"addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":921710019}]},{"addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":1137857729}]}]}} 2024-04-22T21:38:56.642 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T21:38:56.642 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T21:38:56.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T21:38:56.698 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:56 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3097458416' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T21:38:56.698 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:56 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:56.698 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:56 smithi165 ceph-mon[21833]: Deploying daemon node-exporter.a on smithi165 2024-04-22T21:38:56.698 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:56 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3097458416' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T21:38:56.698 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:56 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:56.699 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:56 smithi165 ceph-mon[26549]: Deploying daemon node-exporter.a on smithi165 2024-04-22T21:38:56.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:56 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3097458416' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T21:38:56.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:56 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:56.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:56 smithi188 ceph-mon[22698]: Deploying daemon node-exporter.a on smithi165 2024-04-22T21:38:56.913 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:38:58.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:58 smithi165 ceph-mon[21833]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:58.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:58 smithi165 ceph-mon[26549]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:58.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:58 smithi188 ceph-mon[22698]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:58.744 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:38:58.745 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":60,"fsid":"dccb09a4-00ef-11ef-bc93-c7b262605968","created":"2024-04-22T21:33:22.831640+0000","modified":"2024-04-22T21:38:38.112708+0000","last_up_change":"2024-04-22T21:38:22.352005+0000","last_in_change":"2024-04-22T21:38:05.201912+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:36:26.472879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"315d5f25-fb61-4586-b222-e4463c47ac34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6803","nonce":3819369935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6805","nonce":3819369935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6809","nonce":3819369935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6807","nonce":3819369935}]},"public_addr":"172.21.15.165:6803/3819369935","cluster_addr":"172.21.15.165:6805/3819369935","heartbeat_back_addr":"172.21.15.165:6809/3819369935","heartbeat_front_addr":"172.21.15.165:6807/3819369935","state":["exists","up"]},{"osd":1,"uuid":"8e61383e-0138-4fb4-ac5a-1a9f96406576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6811","nonce":4196345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6813","nonce":4196345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6817","nonce":4196345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6815","nonce":4196345941}]},"public_addr":"172.21.15.165:6811/4196345941","cluster_addr":"172.21.15.165:6813/4196345941","heartbeat_back_addr":"172.21.15.165:6817/4196345941","heartbeat_front_addr":"172.21.15.165:6815/4196345941","state":["exists","up"]},{"osd":2,"uuid":"0b1b573c-17c0-4e85-9f32-e66637e8fe0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6819","nonce":4051532826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6821","nonce":4051532826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6825","nonce":4051532826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6823","nonce":4051532826}]},"public_addr":"172.21.15.165:6819/4051532826","cluster_addr":"172.21.15.165:6821/4051532826","heartbeat_back_addr":"172.21.15.165:6825/4051532826","heartbeat_front_addr":"172.21.15.165:6823/4051532826","state":["exists","up"]},{"osd":3,"uuid":"5a6e8740-aefe-4aa7-a6bc-45a8fe626b24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6827","nonce":1668061591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6829","nonce":1668061591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6833","nonce":1668061591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6831","nonce":1668061591}]},"public_addr":"172.21.15.165:6827/1668061591","cluster_addr":"172.21.15.165:6829/1668061591","heartbeat_back_addr":"172.21.15.165:6833/1668061591","heartbeat_front_addr":"172.21.15.165:6831/1668061591","state":["exists","up"]},{"osd":4,"uuid":"dc54e2dc-af18-42ba-897c-b0e06fafb520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6801","nonce":1586199781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6803","nonce":1586199781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6807","nonce":1586199781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6805","nonce":1586199781}]},"public_addr":"172.21.15.188:6801/1586199781","cluster_addr":"172.21.15.188:6803/1586199781","heartbeat_back_addr":"172.21.15.188:6807/1586199781","heartbeat_front_addr":"172.21.15.188:6805/1586199781","state":["exists","up"]},{"osd":5,"uuid":"75ecfecc-9483-4c4c-98ce-25a89006dcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6809","nonce":1947262752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6811","nonce":1947262752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6815","nonce":1947262752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6813","nonce":1947262752}]},"public_addr":"172.21.15.188:6809/1947262752","cluster_addr":"172.21.15.188:6811/1947262752","heartbeat_back_addr":"172.21.15.188:6815/1947262752","heartbeat_front_addr":"172.21.15.188:6813/1947262752","state":["exists","up"]},{"osd":6,"uuid":"eeceb5bb-7317-422b-9d66-62e099e57b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6817","nonce":1697563430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6819","nonce":1697563430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6823","nonce":1697563430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6821","nonce":1697563430}]},"public_addr":"172.21.15.188:6817/1697563430","cluster_addr":"172.21.15.188:6819/1697563430","heartbeat_back_addr":"172.21.15.188:6823/1697563430","heartbeat_front_addr":"172.21.15.188:6821/1697563430","state":["exists","up"]},{"osd":7,"uuid":"38a0f129-d414-410a-8e8a-37baa9ef6319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6825","nonce":164704082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6827","nonce":164704082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6831","nonce":164704082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6829","nonce":164704082}]},"public_addr":"172.21.15.188:6825/164704082","cluster_addr":"172.21.15.188:6827/164704082","heartbeat_back_addr":"172.21.15.188:6831/164704082","heartbeat_front_addr":"172.21.15.188:6829/164704082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:35:34.798019+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:35:58.341829+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:36:22.899668+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:36:48.166539+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:37:10.294264+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:37:33.510025+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:37:56.516100+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:38:20.714448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:0/229165957":"2024-04-23T21:34:16.115619+0000","172.21.15.165:6801/438332672":"2024-04-23T21:38:38.112659+0000","172.21.15.165:6801/2127003174":"2024-04-23T21:34:16.115619+0000","172.21.15.165:6800/2127003174":"2024-04-23T21:34:16.115619+0000","172.21.15.165:0/1056841091":"2024-04-23T21:38:38.112659+0000","172.21.15.165:0/2427714119":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/2436844333":"2024-04-23T21:38:38.112659+0000","172.21.15.165:6800/4281613920":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/1312953247":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/3661503622":"2024-04-23T21:38:38.112659+0000","172.21.15.165:0/3105631583":"2024-04-23T21:38:38.112659+0000","172.21.15.165:6800/438332672":"2024-04-23T21:38:38.112659+0000","172.21.15.165:0/1483522820":"2024-04-23T21:34:16.115619+0000","172.21.15.165:6801/4281613920":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/3912587609":"2024-04-23T21:34:16.115619+0000","172.21.15.165:0/3416385233":"2024-04-23T21:33:52.472131+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:38:59.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:59 smithi188 ceph-mon[22698]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:59.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:59 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:59.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:38:59 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1730994324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:38:59.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:59 smithi165 ceph-mon[21833]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:59.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:59 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:59.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:38:59 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1730994324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:38:59.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:59 smithi165 ceph-mon[26549]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:38:59.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:59 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:38:59.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:38:59 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1730994324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:38:59.842 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T21:38:59.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T21:39:00.590 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:01.502 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:01 smithi165 ceph-mon[21833]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:01.502 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:01 smithi165 ceph-mon[26549]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:01.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:01 smithi188 ceph-mon[22698]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:02.207 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:01 smithi165 podman[41437]: 2024-04-22 21:39:01.981755141 +0000 UTC m=+3.418549363 container create f4355268ef2fe03a9d2b730d57b32688880f04f5ab74ee7ed3a26d501c7e0192 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T21:39:02.510 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 podman[41437]: 2024-04-22 21:39:02.225975733 +0000 UTC m=+3.662769955 container init f4355268ef2fe03a9d2b730d57b32688880f04f5ab74ee7ed3a26d501c7e0192 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T21:39:02.510 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 podman[41437]: 2024-04-22 21:39:02.229234669 +0000 UTC m=+3.666028893 container start f4355268ef2fe03a9d2b730d57b32688880f04f5ab74ee7ed3a26d501c7e0192 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T21:39:02.510 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.233Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-22T21:39:02.511 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.233Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-22T21:39:02.511 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.234Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-22T21:39:02.511 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.234Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-22T21:39:02.511 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-22T21:39:02.511 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=arp 2024-04-22T21:39:02.511 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-22T21:39:02.512 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-22T21:39:02.512 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-22T21:39:02.512 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-22T21:39:02.512 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-22T21:39:02.512 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-22T21:39:02.512 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-22T21:39:02.512 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-22T21:39:02.513 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=edac 2024-04-22T21:39:02.513 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-22T21:39:02.513 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-22T21:39:02.513 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-22T21:39:02.513 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-22T21:39:02.513 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-22T21:39:02.513 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-22T21:39:02.514 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-22T21:39:02.514 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-22T21:39:02.514 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-22T21:39:02.514 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-22T21:39:02.514 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-22T21:39:02.514 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-22T21:39:02.515 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-22T21:39:02.515 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-22T21:39:02.515 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-22T21:39:02.515 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-22T21:39:02.515 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=os 2024-04-22T21:39:02.515 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-22T21:39:02.515 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-22T21:39:02.516 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-22T21:39:02.516 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-22T21:39:02.516 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-22T21:39:02.516 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-22T21:39:02.516 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=stat 2024-04-22T21:39:02.516 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-22T21:39:02.516 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-22T21:39:02.517 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-22T21:39:02.517 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=time 2024-04-22T21:39:02.517 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-22T21:39:02.517 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=uname 2024-04-22T21:39:02.517 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-22T21:39:02.517 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-22T21:39:02.518 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-22T21:39:02.518 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.235Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-22T21:39:02.518 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a[41589]: ts=2024-04-22T21:39:02.236Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-22T21:39:02.518 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 bash[41437]: f4355268ef2fe03a9d2b730d57b32688880f04f5ab74ee7ed3a26d501c7e0192 2024-04-22T21:39:02.518 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:39:02 smithi165 systemd[1]: Started Ceph node-exporter.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:39:03.381 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:03 smithi188 bash[37957]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-22T21:39:03.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:03 smithi165 ceph-mon[21833]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:03.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:03 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:03.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:03 smithi165 ceph-mon[21833]: Deploying daemon node-exporter.b on smithi188 2024-04-22T21:39:03.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:03 smithi165 ceph-mon[26549]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:03.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:03 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:03.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:03 smithi165 ceph-mon[26549]: Deploying daemon node-exporter.b on smithi188 2024-04-22T21:39:03.582 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:39:03.582 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":60,"fsid":"dccb09a4-00ef-11ef-bc93-c7b262605968","created":"2024-04-22T21:33:22.831640+0000","modified":"2024-04-22T21:38:38.112708+0000","last_up_change":"2024-04-22T21:38:22.352005+0000","last_in_change":"2024-04-22T21:38:05.201912+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T21:36:26.472879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"315d5f25-fb61-4586-b222-e4463c47ac34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6803","nonce":3819369935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6805","nonce":3819369935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6809","nonce":3819369935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":3819369935},{"type":"v1","addr":"172.21.15.165:6807","nonce":3819369935}]},"public_addr":"172.21.15.165:6803/3819369935","cluster_addr":"172.21.15.165:6805/3819369935","heartbeat_back_addr":"172.21.15.165:6809/3819369935","heartbeat_front_addr":"172.21.15.165:6807/3819369935","state":["exists","up"]},{"osd":1,"uuid":"8e61383e-0138-4fb4-ac5a-1a9f96406576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6811","nonce":4196345941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6813","nonce":4196345941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6817","nonce":4196345941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":4196345941},{"type":"v1","addr":"172.21.15.165:6815","nonce":4196345941}]},"public_addr":"172.21.15.165:6811/4196345941","cluster_addr":"172.21.15.165:6813/4196345941","heartbeat_back_addr":"172.21.15.165:6817/4196345941","heartbeat_front_addr":"172.21.15.165:6815/4196345941","state":["exists","up"]},{"osd":2,"uuid":"0b1b573c-17c0-4e85-9f32-e66637e8fe0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6819","nonce":4051532826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6821","nonce":4051532826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6825","nonce":4051532826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":4051532826},{"type":"v1","addr":"172.21.15.165:6823","nonce":4051532826}]},"public_addr":"172.21.15.165:6819/4051532826","cluster_addr":"172.21.15.165:6821/4051532826","heartbeat_back_addr":"172.21.15.165:6825/4051532826","heartbeat_front_addr":"172.21.15.165:6823/4051532826","state":["exists","up"]},{"osd":3,"uuid":"5a6e8740-aefe-4aa7-a6bc-45a8fe626b24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6827","nonce":1668061591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6829","nonce":1668061591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6833","nonce":1668061591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1668061591},{"type":"v1","addr":"172.21.15.165:6831","nonce":1668061591}]},"public_addr":"172.21.15.165:6827/1668061591","cluster_addr":"172.21.15.165:6829/1668061591","heartbeat_back_addr":"172.21.15.165:6833/1668061591","heartbeat_front_addr":"172.21.15.165:6831/1668061591","state":["exists","up"]},{"osd":4,"uuid":"dc54e2dc-af18-42ba-897c-b0e06fafb520","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6801","nonce":1586199781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6803","nonce":1586199781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6807","nonce":1586199781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":1586199781},{"type":"v1","addr":"172.21.15.188:6805","nonce":1586199781}]},"public_addr":"172.21.15.188:6801/1586199781","cluster_addr":"172.21.15.188:6803/1586199781","heartbeat_back_addr":"172.21.15.188:6807/1586199781","heartbeat_front_addr":"172.21.15.188:6805/1586199781","state":["exists","up"]},{"osd":5,"uuid":"75ecfecc-9483-4c4c-98ce-25a89006dcd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6809","nonce":1947262752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6811","nonce":1947262752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6815","nonce":1947262752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1947262752},{"type":"v1","addr":"172.21.15.188:6813","nonce":1947262752}]},"public_addr":"172.21.15.188:6809/1947262752","cluster_addr":"172.21.15.188:6811/1947262752","heartbeat_back_addr":"172.21.15.188:6815/1947262752","heartbeat_front_addr":"172.21.15.188:6813/1947262752","state":["exists","up"]},{"osd":6,"uuid":"eeceb5bb-7317-422b-9d66-62e099e57b49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6817","nonce":1697563430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6819","nonce":1697563430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6823","nonce":1697563430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":1697563430},{"type":"v1","addr":"172.21.15.188:6821","nonce":1697563430}]},"public_addr":"172.21.15.188:6817/1697563430","cluster_addr":"172.21.15.188:6819/1697563430","heartbeat_back_addr":"172.21.15.188:6823/1697563430","heartbeat_front_addr":"172.21.15.188:6821/1697563430","state":["exists","up"]},{"osd":7,"uuid":"38a0f129-d414-410a-8e8a-37baa9ef6319","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6825","nonce":164704082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6827","nonce":164704082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6830","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6831","nonce":164704082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6828","nonce":164704082},{"type":"v1","addr":"172.21.15.188:6829","nonce":164704082}]},"public_addr":"172.21.15.188:6825/164704082","cluster_addr":"172.21.15.188:6827/164704082","heartbeat_back_addr":"172.21.15.188:6831/164704082","heartbeat_front_addr":"172.21.15.188:6829/164704082","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:35:34.798019+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:35:58.341829+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:36:22.899668+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:36:48.166539+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:37:10.294264+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:37:33.510025+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:37:56.516100+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T21:38:20.714448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.165:0/229165957":"2024-04-23T21:34:16.115619+0000","172.21.15.165:6801/438332672":"2024-04-23T21:38:38.112659+0000","172.21.15.165:6801/2127003174":"2024-04-23T21:34:16.115619+0000","172.21.15.165:6800/2127003174":"2024-04-23T21:34:16.115619+0000","172.21.15.165:0/1056841091":"2024-04-23T21:38:38.112659+0000","172.21.15.165:0/2427714119":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/2436844333":"2024-04-23T21:38:38.112659+0000","172.21.15.165:6800/4281613920":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/1312953247":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/3661503622":"2024-04-23T21:38:38.112659+0000","172.21.15.165:0/3105631583":"2024-04-23T21:38:38.112659+0000","172.21.15.165:6800/438332672":"2024-04-23T21:38:38.112659+0000","172.21.15.165:0/1483522820":"2024-04-23T21:34:16.115619+0000","172.21.15.165:6801/4281613920":"2024-04-23T21:33:52.472131+0000","172.21.15.165:0/3912587609":"2024-04-23T21:34:16.115619+0000","172.21.15.165:0/3416385233":"2024-04-23T21:33:52.472131+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T21:39:03.643 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:03 smithi188 ceph-mon[22698]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:03.644 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:03 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:03.644 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:03 smithi188 ceph-mon[22698]: Deploying daemon node-exporter.b on smithi188 2024-04-22T21:39:04.007 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:03 smithi188 bash[37957]: Getting image source signatures 2024-04-22T21:39:04.007 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:03 smithi188 bash[37957]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-22T21:39:04.007 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:03 smithi188 bash[37957]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-22T21:39:04.007 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:03 smithi188 bash[37957]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-22T21:39:04.284 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:39:03 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:03] "GET /metrics HTTP/1.1" 200 191534 "" "Prometheus/2.33.4" 2024-04-22T21:39:04.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T21:39:04.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T21:39:04.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T21:39:04.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T21:39:04.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T21:39:04.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T21:39:04.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T21:39:04.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T21:39:04.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:04 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2516216250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:39:04.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:04 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/2516216250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:39:04.734 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:04.738 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:04.738 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:04.742 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:04.742 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:04.742 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:04.742 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:04.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:04 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/2516216250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T21:39:04.760 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:05.500 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:05 smithi165 ceph-mon[21833]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:05.500 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:05 smithi165 ceph-mon[26549]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:05.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:05 smithi188 ceph-mon[22698]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:06.236 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:39:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:39:06.237 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 bash[37957]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-22T21:39:06.507 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 bash[37957]: Writing manifest to image destination 2024-04-22T21:39:06.826 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 podman[37957]: 2024-04-22 21:39:06.477517288 +0000 UTC m=+3.329454939 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-22T21:39:06.826 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 podman[37957]: 2024-04-22 21:39:06.644296044 +0000 UTC m=+3.496233710 container create 211a953ad0ee2314903b007c3deff814eb0f168e890df47cd6adf2a56adfb654 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T21:39:07.257 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 podman[37957]: 2024-04-22 21:39:06.845142043 +0000 UTC m=+3.697079678 container init 211a953ad0ee2314903b007c3deff814eb0f168e890df47cd6adf2a56adfb654 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T21:39:07.257 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 podman[37957]: 2024-04-22 21:39:06.84824144 +0000 UTC m=+3.700179057 container start 211a953ad0ee2314903b007c3deff814eb0f168e890df47cd6adf2a56adfb654 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T21:39:07.257 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.852Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-22T21:39:07.257 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.852Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=arp 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-22T21:39:07.258 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=edac 2024-04-22T21:39:07.259 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-22T21:39:07.259 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-22T21:39:07.259 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-22T21:39:07.259 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-22T21:39:07.259 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-22T21:39:07.259 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-22T21:39:07.259 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=os 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-22T21:39:07.260 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=stat 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=time 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=uname 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.853Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.854Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b[38007]: ts=2024-04-22T21:39:06.854Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 bash[37957]: 211a953ad0ee2314903b007c3deff814eb0f168e890df47cd6adf2a56adfb654 2024-04-22T21:39:07.261 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:39:06 smithi188 systemd[1]: Started Ceph node-exporter.b for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:39:08.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[21833]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:08.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:08.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:39:08.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:39:08.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:39:08.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[26549]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:08.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:08.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:39:08.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:39:08.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:07 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:39:08.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:07 smithi188 ceph-mon[22698]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:08.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:07 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:08.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:07 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T21:39:08.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:07 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:39:08.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:07 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:39:09.684 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:09 smithi188 ceph-mon[22698]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:09.684 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:09 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:09.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:09 smithi165 ceph-mon[21833]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:09.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:09 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:09.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:09 smithi165 ceph-mon[26549]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:09.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:09 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:10.947 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:10 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:10.947 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:10 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:11.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:10 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:12.002 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:11 smithi165 ceph-mon[21833]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:12.002 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:11 smithi165 ceph-mon[26549]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:12.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:11 smithi188 ceph-mon[22698]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:13.720 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:13 smithi165 ceph-mon[26549]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:13.720 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:13 smithi165 ceph-mon[21833]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:13.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:13 smithi188 ceph-mon[22698]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:14.057 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:39:13 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:13] "GET /metrics HTTP/1.1" 200 191513 "" "Prometheus/2.33.4" 2024-04-22T21:39:14.201 INFO:teuthology.orchestra.run.smithi165.stdout:236223201291 2024-04-22T21:39:14.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T21:39:15.557 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:15 smithi165 ceph-mon[21833]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:15.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:15 smithi165 ceph-mon[26549]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:15.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:15 smithi188 ceph-mon[22698]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:16.015 INFO:teuthology.orchestra.run.smithi165.stdout:167503724566 2024-04-22T21:39:16.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T21:39:16.257 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:39:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:39:17.545 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:17 smithi165 ceph-mon[21833]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:17.545 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:17 smithi165 ceph-mon[26549]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:17.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:17 smithi188 ceph-mon[22698]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:18.753 INFO:teuthology.orchestra.run.smithi165.stdout:34359738413 2024-04-22T21:39:18.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T21:39:19.626 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:19 smithi165 ceph-mon[21833]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:19.627 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:19 smithi165 ceph-mon[26549]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:19.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:19 smithi188 ceph-mon[22698]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:19.952 INFO:teuthology.orchestra.run.smithi165.stdout:55834574890 2024-04-22T21:39:19.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T21:39:20.625 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:20.626 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:21.132 INFO:teuthology.orchestra.run.smithi165.stdout:111669149728 2024-04-22T21:39:21.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T21:39:21.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:21 smithi188 ceph-mon[22698]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:21.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:21 smithi165 ceph-mon[21833]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:21.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:21 smithi165 ceph-mon[26549]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:22.357 INFO:teuthology.orchestra.run.smithi165.stdout:77309411366 2024-04-22T21:39:22.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T21:39:22.513 INFO:teuthology.orchestra.run.smithi165.stdout:201863462929 2024-04-22T21:39:22.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T21:39:23.484 INFO:teuthology.orchestra.run.smithi165.stdout:133143986204 2024-04-22T21:39:23.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T21:39:23.628 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:23 smithi165 ceph-mon[21833]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:23.629 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:23 smithi165 ceph-mon[26549]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:23.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:23 smithi188 ceph-mon[22698]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:24.308 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:39:23 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:23] "GET /metrics HTTP/1.1" 200 191510 "" "Prometheus/2.33.4" 2024-04-22T21:39:24.512 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:24.512 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:24.512 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:25.032 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:25.033 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:25.033 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:25.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:25 smithi188 ceph-mon[22698]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:25.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:25 smithi165 ceph-mon[21833]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:25.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:25 smithi165 ceph-mon[26549]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:26.257 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:39:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:39:27.632 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:27 smithi165 ceph-mon[26549]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:27.632 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:27 smithi165 ceph-mon[21833]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:27.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:27 smithi188 ceph-mon[22698]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:28.560 INFO:teuthology.orchestra.run.smithi165.stdout:236223201294 2024-04-22T21:39:28.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:28 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1473904414' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T21:39:28.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:28 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1473904414' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T21:39:28.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:28 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1473904414' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T21:39:29.255 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201291 got 236223201294 for osd.7 2024-04-22T21:39:29.255 DEBUG:teuthology.parallel:result is None 2024-04-22T21:39:29.483 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:29 smithi165 ceph-mon[21833]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:29.484 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:29 smithi165 ceph-mon[26549]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:29.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:29 smithi188 ceph-mon[22698]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:31.538 INFO:teuthology.orchestra.run.smithi165.stdout:167503724569 2024-04-22T21:39:31.538 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:31 smithi165 ceph-mon[21833]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:31.538 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:31 smithi165 ceph-mon[26549]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:31.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:31 smithi188 ceph-mon[22698]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:32.557 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:32 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1407307177' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T21:39:32.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:32 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1407307177' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T21:39:32.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:32 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1407307177' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T21:39:33.388 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724566 got 167503724569 for osd.5 2024-04-22T21:39:33.388 DEBUG:teuthology.parallel:result is None 2024-04-22T21:39:33.679 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:33 smithi165 ceph-mon[26549]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:33.680 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:33 smithi165 ceph-mon[21833]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:33.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:33 smithi188 ceph-mon[22698]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:34.005 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:39:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:33] "GET /metrics HTTP/1.1" 200 191510 "" "Prometheus/2.33.4" 2024-04-22T21:39:34.557 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:34 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1253139217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T21:39:34.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:34 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1253139217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T21:39:34.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:34 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1253139217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T21:39:34.940 INFO:teuthology.orchestra.run.smithi165.stdout:201863462932 2024-04-22T21:39:35.557 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:35 smithi165 ceph-mon[21833]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:35.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:35 smithi165 ceph-mon[26549]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:35.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:35 smithi188 ceph-mon[22698]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:36.257 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:39:35 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:39:36.694 INFO:teuthology.orchestra.run.smithi165.stdout:111669149732 2024-04-22T21:39:37.639 INFO:teuthology.orchestra.run.smithi165.stdout:77309411369 2024-04-22T21:39:37.698 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:37 smithi165 ceph-mon[21833]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:37.698 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:37 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/4036551084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T21:39:37.698 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:37 smithi165 ceph-mon[26549]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:37.698 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:37 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/4036551084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T21:39:37.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:37 smithi188 ceph-mon[22698]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:37.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:37 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/4036551084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T21:39:38.557 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:38 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/656682976' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T21:39:38.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:38 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1058271881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T21:39:38.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:39:38.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:39:38.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:38 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/656682976' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T21:39:38.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:38 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1058271881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T21:39:38.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:39:38.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:39:38.625 INFO:teuthology.orchestra.run.smithi165.stdout:133143986207 2024-04-22T21:39:38.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:38 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/656682976' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T21:39:38.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:38 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1058271881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T21:39:38.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:39:38.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:39:39.403 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986204 got 133143986207 for osd.4 2024-04-22T21:39:39.403 DEBUG:teuthology.parallel:result is None 2024-04-22T21:39:39.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:39 smithi188 ceph-mon[22698]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:39.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:39 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/4045363112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T21:39:39.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:39 smithi165 ceph-mon[21833]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:39.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:39 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/4045363112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T21:39:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:39 smithi165 ceph-mon[26549]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:39 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/4045363112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T21:39:39.864 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411369 for osd.2 2024-04-22T21:39:39.864 DEBUG:teuthology.parallel:result is None 2024-04-22T21:39:40.127 INFO:teuthology.orchestra.run.smithi165.stdout:34359738417 2024-04-22T21:39:40.223 INFO:teuthology.orchestra.run.smithi165.stdout:55834574894 2024-04-22T21:39:40.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:40 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/959173293' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T21:39:40.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:40 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/959173293' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T21:39:40.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:40 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/959173293' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T21:39:41.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149728 got 111669149732 for osd.3 2024-04-22T21:39:41.353 DEBUG:teuthology.parallel:result is None 2024-04-22T21:39:41.738 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:41 smithi165 ceph-mon[21833]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:41.739 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:41 smithi165 ceph-mon[26549]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:41.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:41 smithi188 ceph-mon[22698]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:41.765 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462929 got 201863462932 for osd.6 2024-04-22T21:39:41.765 DEBUG:teuthology.parallel:result is None 2024-04-22T21:39:42.430 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574890 got 55834574894 for osd.1 2024-04-22T21:39:42.431 DEBUG:teuthology.parallel:result is None 2024-04-22T21:39:42.597 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738417 for osd.0 2024-04-22T21:39:42.597 DEBUG:teuthology.parallel:result is None 2024-04-22T21:39:42.598 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T21:39:42.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T21:39:43.039 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:43.418 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:43 smithi165 ceph-mon[21833]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:43.418 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:43 smithi165 ceph-mon[26549]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:43.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:43 smithi188 ceph-mon[22698]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:44.057 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:39:43 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:43] "GET /metrics HTTP/1.1" 200 191507 "" "Prometheus/2.33.4" 2024-04-22T21:39:44.557 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:44 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:44.557 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:44 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:44.558 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:44 smithi165 ceph-mon[21833]: Deploying daemon alertmanager.a on smithi165 2024-04-22T21:39:44.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:44 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:44.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:44 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:44.558 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:44 smithi165 ceph-mon[26549]: Deploying daemon alertmanager.a on smithi165 2024-04-22T21:39:44.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:44 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:44.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:44 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:44.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:44 smithi188 ceph-mon[22698]: Deploying daemon alertmanager.a on smithi165 2024-04-22T21:39:45.370 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:39:45.372 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-04-22T21:39:46.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:45 smithi188 ceph-mon[22698]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:46.007 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:39:45 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:39:46.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:45 smithi165 ceph-mon[21833]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:46.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:45 smithi165 ceph-mon[26549]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:46.424 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-22T21:39:44.385176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48968,"kb_used_data":6152,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912248,"statfs":{"total":767960285184,"available":767910141952,"internally_reserved":0,"allocated":6299648,"data_stored":3510946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002140"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T21:38:38.456370+0000","last_change":"2024-04-22T21:38:26.371115+0000","last_active":"2024-04-22T21:38:38.456370+0000","last_peered":"2024-04-22T21:38:38.456370+0000","last_clean":"2024-04-22T21:38:38.456370+0000","last_became_active":"2024-04-22T21:38:26.370656+0000","last_became_peered":"2024-04-22T21:38:26.370656+0000","last_unstale":"2024-04-22T21:38:38.456370+0000","last_undegraded":"2024-04-22T21:38:38.456370+0000","last_fullsized":"2024-04-22T21:38:38.456370+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T21:36:26.843852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T21:36:26.843852+0000","last_clean_scrub_stamp":"2024-04-22T21:36:26.843852+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:01:32.151600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45100000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.59499999999999997}]},{"osd":2,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.61099999999999999}]},{"osd":4,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.67700000000000005}]},{"osd":5,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.63700000000000001}]},{"osd":6,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.621}]}]},{"osd":6,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6160,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738992,"statfs":{"total":95995035648,"available":95988727808,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.496},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51400000000000001}]},{"osd":3,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.57799999999999996}]},{"osd":4,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.55900000000000005}]},{"osd":5,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.61699999999999999}]},{"osd":7,"last update":"Mon Apr 22 21:39:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.70199999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36199999999999999,"15min":0.34899999999999998},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.377,"15min":0.36399999999999999},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54900000000000004,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Apr 22 21:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.39300000000000002,"15min":0.39900000000000002},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46700000000000003,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.29899999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Apr 22 21:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.46899999999999997,"15min":0.48099999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.50800000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.48999999999999999,"15min":0.496},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.378}]},{"osd":4,"last update":"Mon Apr 22 21:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.624},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.75,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57699999999999996}]},{"osd":5,"last update":"Mon Apr 22 21:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.72499999999999998,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Apr 22 21:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.64100000000000001}]},{"osd":7,"last update":"Mon Apr 22 21:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6924,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738228,"statfs":{"total":95995035648,"available":95987945472,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 21:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.32100000000000001},"min":{"1min":0.25,"5min":0.217,"15min":0.217},"max":{"1min":0.47099999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.34799999999999998,"15min":0.32300000000000001},"min":{"1min":0.29399999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Apr 22 21:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.39100000000000001},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.5,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.379},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.40100000000000002,"15min":0.40000000000000002},"min":{"1min":0.28799999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.46100000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.36699999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:38:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.432,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46500000000000002,"15min":0.46700000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.58299999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Apr 22 21:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83899999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.626},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67300000000000004,"5min":1.054,"15min":1.054},"last":0.70799999999999996}]},{"osd":5,"last update":"Mon Apr 22 21:39:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79800000000000004,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.44500000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.75,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.75}]},{"osd":6,"last update":"Mon Apr 22 21:39:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Apr 22 21:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.78100000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.26500000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46899999999999997,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.29999999999999999}]},{"osd":1,"last update":"Mon Apr 22 21:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57399999999999995,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.48599999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:38:52 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.56799999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.35099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.623,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Apr 22 21:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57099999999999995,"15min":0.57399999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70999999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.72699999999999998,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.47799999999999998}]},{"osd":5,"last update":"Mon Apr 22 21:38:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Mon Apr 22 21:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60999999999999999}]},{"osd":7,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60399999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:38:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41499999999999998,"15min":0.41799999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.53700000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.374},{"interface":"front","average":{"1min":0.439,"5min":0.42499999999999999,"15min":0.42299999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60499999999999998,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.44}]},{"osd":1,"last update":"Mon Apr 22 21:38:53 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.53700000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.75,"15min":0.75},"last":0.45900000000000002}]},{"osd":2,"last update":"Mon Apr 22 21:38:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67600000000000005,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.438,"15min":0.44},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60199999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Apr 22 21:39:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.70699999999999996}]},{"osd":5,"last update":"Mon Apr 22 21:38:37 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.74299999999999999}]},{"osd":6,"last update":"Mon Apr 22 21:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Apr 22 21:39:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.79900000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39300000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Apr 22 21:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.438},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Apr 22 21:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70699999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.379},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.45200000000000001}]},{"osd":3,"last update":"Mon Apr 22 21:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.52900000000000003}]},{"osd":5,"last update":"Mon Apr 22 21:38:35 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.503}]},{"osd":6,"last update":"Mon Apr 22 21:39:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51900000000000002}]},{"osd":7,"last update":"Mon Apr 22 21:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.48999999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.34899999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.51700000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.435},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.504,"15min":0.498},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.54000000000000004}]},{"osd":2,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.375,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69099999999999995,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47599999999999998}]},{"osd":3,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56000000000000005,"15min":0.55500000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.41399999999999998,"15min":0.40200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.442,"15min":0.43099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.34100000000000003}]},{"osd":6,"last update":"Mon Apr 22 21:38:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.38700000000000001}]},{"osd":7,"last update":"Mon Apr 22 21:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T21:39:46.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T21:39:46.642 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:47.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:46 smithi188 ceph-mon[22698]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:39:47.012 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:46 smithi165 ceph-mon[21833]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:39:47.012 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:46 smithi165 ceph-mon[26549]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:39:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:47 smithi188 ceph-mon[22698]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:48.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:47 smithi165 ceph-mon[21833]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:48.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:47 smithi165 ceph-mon[26549]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:49.535 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:39:49.536 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2024-04-22T21:39:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:49 smithi188 ceph-mon[22698]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:50.021 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:49 smithi165 ceph-mon[21833]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:50.022 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:49 smithi165 ceph-mon[26549]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:50.836 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:50 smithi165 ceph-mon[21833]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:39:50.836 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:50 smithi165 ceph-mon[26549]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:39:50.837 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-22T21:39:48.386032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48968,"kb_used_data":6152,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912248,"statfs":{"total":767960285184,"available":767910141952,"internally_reserved":0,"allocated":6299648,"data_stored":3510946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002384"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-22T21:38:38.456370+0000","last_change":"2024-04-22T21:38:26.371115+0000","last_active":"2024-04-22T21:38:38.456370+0000","last_peered":"2024-04-22T21:38:38.456370+0000","last_clean":"2024-04-22T21:38:38.456370+0000","last_became_active":"2024-04-22T21:38:26.370656+0000","last_became_peered":"2024-04-22T21:38:26.370656+0000","last_unstale":"2024-04-22T21:38:38.456370+0000","last_undegraded":"2024-04-22T21:38:38.456370+0000","last_fullsized":"2024-04-22T21:38:38.456370+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T21:36:26.843852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T21:36:26.843852+0000","last_clean_scrub_stamp":"2024-04-22T21:36:26.843852+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:01:32.151600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.63500000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.58899999999999997}]},{"osd":2,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.65600000000000003}]},{"osd":3,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.42499999999999999}]},{"osd":5,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.44800000000000001}]},{"osd":6,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.48599999999999999}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6160,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738992,"statfs":{"total":95995035648,"available":95988727808,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.52800000000000002}]},{"osd":1,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.48099999999999998}]},{"osd":5,"last update":"Mon Apr 22 21:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Apr 22 21:39:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.67100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.36199999999999999,"15min":0.34899999999999998},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.59299999999999997,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.377,"15min":0.36399999999999999},"min":{"1min":0.27300000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54900000000000004,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.39500000000000002}]},{"osd":2,"last update":"Mon Apr 22 21:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.39300000000000002,"15min":0.39900000000000002},"min":{"1min":0.27400000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.46700000000000003,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.29899999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Apr 22 21:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.46899999999999997,"15min":0.48099999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.50800000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.48999999999999999,"15min":0.496},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.58799999999999997}]},{"osd":4,"last update":"Mon Apr 22 21:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.75,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.63800000000000001}]},{"osd":5,"last update":"Mon Apr 22 21:39:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.72499999999999998,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.73199999999999998}]},{"osd":6,"last update":"Mon Apr 22 21:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.77700000000000002}]},{"osd":7,"last update":"Mon Apr 22 21:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.747},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.64800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6924,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738228,"statfs":{"total":95995035648,"available":95987945472,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 21:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33200000000000002,"15min":0.32100000000000001},"min":{"1min":0.25,"5min":0.217,"15min":0.217},"max":{"1min":0.47099999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.34799999999999998,"15min":0.32300000000000001},"min":{"1min":0.29399999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Apr 22 21:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.39100000000000001},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.5,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.40100000000000002,"15min":0.40000000000000002},"min":{"1min":0.28799999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.46100000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.33900000000000002}]},{"osd":3,"last update":"Mon Apr 22 21:38:55 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.432,"15min":0.436},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46500000000000002,"15min":0.46700000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.58299999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Apr 22 21:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83899999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.67300000000000004,"5min":1.054,"15min":1.054},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Apr 22 21:39:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79800000000000004,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.44500000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.75,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.59999999999999998}]},{"osd":6,"last update":"Mon Apr 22 21:39:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.58899999999999997}]},{"osd":7,"last update":"Mon Apr 22 21:39:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.64900000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.26500000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46899999999999997,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.44},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Apr 22 21:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57399999999999995,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.40400000000000003,"15min":0.40200000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.48599999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.499}]},{"osd":3,"last update":"Mon Apr 22 21:38:52 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.56799999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47699999999999998,"15min":0.47999999999999998},"min":{"1min":0.35099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.623,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.51400000000000001}]},{"osd":4,"last update":"Mon Apr 22 21:39:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57099999999999995,"15min":0.57399999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70999999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.72699999999999998,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.80700000000000005}]},{"osd":5,"last update":"Mon Apr 22 21:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81799999999999995,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.441,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.76300000000000001,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.76300000000000001}]},{"osd":6,"last update":"Mon Apr 22 21:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.79800000000000004}]},{"osd":7,"last update":"Mon Apr 22 21:39:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.72699999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:38:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.41499999999999998,"15min":0.41799999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.53700000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.312},{"interface":"front","average":{"1min":0.439,"5min":0.42499999999999999,"15min":0.42299999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.60499999999999998,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Apr 22 21:38:53 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.53700000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.75,"15min":0.75},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Apr 22 21:38:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67600000000000005,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.438,"15min":0.44},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60199999999999998,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.28599999999999998}]},{"osd":4,"last update":"Mon Apr 22 21:39:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58399999999999996,"15min":0.58699999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Apr 22 21:39:42 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57399999999999995,"15min":0.56899999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78000000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.45200000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.441}]},{"osd":6,"last update":"Mon Apr 22 21:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.54200000000000004}]},{"osd":7,"last update":"Mon Apr 22 21:39:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.499},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.504}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39300000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.47899999999999998}]},{"osd":1,"last update":"Mon Apr 22 21:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Apr 22 21:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70699999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54300000000000004,"15min":0.54200000000000004},"min":{"1min":0.40000000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.61699999999999999}]},{"osd":3,"last update":"Mon Apr 22 21:39:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.377,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57699999999999996}]},{"osd":5,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.38800000000000001,"15min":0.378},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42099999999999999,"15min":0.40899999999999997},"min":{"1min":0.316,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Apr 22 21:39:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.77300000000000002}]},{"osd":7,"last update":"Mon Apr 22 21:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.754},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.71899999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.62},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.34899999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.51700000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.504,"15min":0.498},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.50900000000000001,"15min":0.504},"min":{"1min":0.375,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69099999999999995,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.52400000000000002}]},{"osd":3,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56000000000000005,"15min":0.55500000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Apr 22 21:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.41399999999999998,"15min":0.40200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.441},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.442,"15min":0.43099999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.42899999999999999}]},{"osd":6,"last update":"Mon Apr 22 21:38:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.41499999999999998}]},{"osd":7,"last update":"Mon Apr 22 21:39:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.371},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T21:39:50.838 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T21:39:50.838 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T21:39:50.838 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T21:39:50.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T21:39:51.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:50 smithi188 ceph-mon[22698]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T21:39:51.900 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:51 smithi165 ceph-mon[21833]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:51.900 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:51 smithi165 ceph-mon[26549]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:52.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:51 smithi188 ceph-mon[22698]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:52.827 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T21:39:53.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:53 smithi188 ceph-mon[22698]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:53.788 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:53 smithi165 ceph-mon[21833]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:53.789 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:53 smithi165 ceph-mon[26549]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:54.057 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:39:53 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:53] "GET /metrics HTTP/1.1" 200 191519 "" "Prometheus/2.33.4" 2024-04-22T21:39:54.965 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-04-22T21:39:54.965 INFO:teuthology.orchestra.run.smithi165.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T21:39:55.668 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:55 smithi165 ceph-mon[21833]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:55.669 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:55 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3904085535' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T21:39:55.669 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:55 smithi165 ceph-mon[26549]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:55.669 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:55 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3904085535' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T21:39:55.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:55 smithi188 ceph-mon[22698]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:55.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:55 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3904085535' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T21:39:56.017 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T21:39:56.017 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T21:39:56.017 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:39:56.027 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi165.front.sepia.ceph.com 2024-04-22T21:39:56.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-04-22T21:39:56.257 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:39:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:39:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:39:56.589 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:56.589 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:56.589 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:56.589 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:39:56.589 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:39:56.590 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:56.590 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[21833]: Deploying daemon grafana.a on smithi188 2024-04-22T21:39:56.590 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:56.590 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:56.590 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:56.590 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:39:56.590 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:39:56.590 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:56.590 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:56 smithi165 ceph-mon[26549]: Deploying daemon grafana.a on smithi188 2024-04-22T21:39:57.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:56 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:57.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:56 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:57.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:56 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:57.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:56 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:39:57.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:56 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T21:39:57.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:56 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:57.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:56 smithi188 ceph-mon[22698]: Deploying daemon grafana.a on smithi188 2024-04-22T21:39:58.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:57 smithi188 ceph-mon[22698]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:58.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:57 smithi165 ceph-mon[21833]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:58.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:57 smithi165 ceph-mon[26549]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:58.669 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:39:58.678 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi165.front.sepia.ceph.com 2024-04-22T21:39:58.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-22T21:39:58.985 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:58 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' 2024-04-22T21:39:58.985 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:58 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:58.985 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:58 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' 2024-04-22T21:39:58.985 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:58 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:59.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:58 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' 2024-04-22T21:39:59.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:58 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:39:59.933 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:39:59 smithi165 ceph-mon[26549]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:39:59.933 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:39:59 smithi165 ceph-mon[21833]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:00.157 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:39:59 smithi188 ceph-mon[22698]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:01.169 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:00 smithi188 ceph-mon[22698]: overall HEALTH_OK 2024-04-22T21:40:01.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:00 smithi165 ceph-mon[21833]: overall HEALTH_OK 2024-04-22T21:40:01.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:00 smithi165 ceph-mon[26549]: overall HEALTH_OK 2024-04-22T21:40:01.934 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-04-22T21:40:01.934 INFO:teuthology.orchestra.run.smithi165.stdout: "id": "d2111914-a014-4462-a1c8-405f0fb86365", 2024-04-22T21:40:01.934 INFO:teuthology.orchestra.run.smithi165.stdout: "name": "r", 2024-04-22T21:40:01.935 INFO:teuthology.orchestra.run.smithi165.stdout: "current_period": "fa325460-e75f-4ffa-865f-6b75c84fd264", 2024-04-22T21:40:01.935 INFO:teuthology.orchestra.run.smithi165.stdout: "epoch": 1 2024-04-22T21:40:01.935 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-04-22T21:40:02.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:01 smithi188 ceph-mon[22698]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:02.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:01 smithi188 ceph-mon[22698]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T21:40:02.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:01 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/804401143' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T21:40:02.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:01 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T21:40:02.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:01 smithi165 ceph-mon[21833]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:02.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:01 smithi165 ceph-mon[21833]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T21:40:02.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:01 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/804401143' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T21:40:02.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:01 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T21:40:02.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:01 smithi165 ceph-mon[26549]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:02.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:01 smithi165 ceph-mon[26549]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T21:40:02.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:01 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/804401143' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T21:40:02.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:01 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T21:40:02.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-22T21:40:03.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:02 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-22T21:40:03.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:02 smithi165 ceph-mon[21833]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T21:40:03.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:02 smithi165 ceph-mon[21833]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T21:40:03.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:02 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-22T21:40:03.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:02 smithi165 ceph-mon[26549]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T21:40:03.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:02 smithi165 ceph-mon[26549]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T21:40:03.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:02 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-22T21:40:03.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:02 smithi188 ceph-mon[22698]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T21:40:03.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:02 smithi188 ceph-mon[22698]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T21:40:04.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:03 smithi165 ceph-mon[21833]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:04.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:03 smithi165 ceph-mon[26549]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:04.059 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:40:03 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:03] "GET /metrics HTTP/1.1" 200 191519 "" "Prometheus/2.33.4" 2024-04-22T21:40:04.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:03 smithi188 ceph-mon[22698]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:04.761 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-04-22T21:40:04.761 INFO:teuthology.orchestra.run.smithi165.stdout: "id": "4a2dd0f3-a400-42cb-a3a5-2b9589a31b4b", 2024-04-22T21:40:04.761 INFO:teuthology.orchestra.run.smithi165.stdout: "name": "default", 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "api_name": "default", 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "is_master": "true", 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "endpoints": [], 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "hostnames": [], 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "hostnames_s3website": [], 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "master_zone": "", 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "zones": [], 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "placement_targets": [], 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "default_placement": "", 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "realm_id": "d2111914-a014-4462-a1c8-405f0fb86365", 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "sync_policy": { 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "groups": [] 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:04.762 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-04-22T21:40:05.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-22T21:40:05.950 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:05 smithi188 ceph-mon[22698]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:06.053 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:05 smithi165 ceph-mon[21833]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:06.053 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:05 smithi165 ceph-mon[26549]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:06.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:40:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:40:07.335 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "id": "32bf1395-a8ce-4d2c-bb1b-b6b6f5c218ff", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "name": "z", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "domain_root": "z.rgw.meta:root", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "control_pool": "z.rgw.control", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "log_pool": "z.rgw.log", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-22T21:40:07.336 INFO:teuthology.orchestra.run.smithi165.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "otp_pool": "z.rgw.otp", 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "system_key": { 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "access_key": "", 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "secret_key": "" 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "placement_pools": [ 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "key": "default-placement", 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "val": { 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "storage_classes": { 2024-04-22T21:40:07.337 INFO:teuthology.orchestra.run.smithi165.stdout: "STANDARD": { 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: "index_type": 0 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: "realm_id": "d2111914-a014-4462-a1c8-405f0fb86365", 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-22T21:40:07.338 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-04-22T21:40:07.998 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:07 smithi165 ceph-mon[21833]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-22T21:40:07.998 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:07 smithi165 ceph-mon[26549]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-22T21:40:08.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-22T21:40:08.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:07 smithi188 ceph-mon[22698]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-22T21:40:08.257 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 systemd[1]: Starting Ceph grafana.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:40:08.553 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 podman[38702]: 2024-04-22 21:40:08.201951222 +0000 UTC m=+0.017413558 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-22T21:40:08.554 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 podman[38702]: 2024-04-22 21:40:08.336541718 +0000 UTC m=+0.152004095 container create 1184bc13bc40423f8cd8b88a30e37d23ac5f8eefbe93eb6a685bcc42fb8f6d5a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a, name=ubi8, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.openshift.expose-services=, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public) 2024-04-22T21:40:08.554 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 podman[38702]: 2024-04-22 21:40:08.572570446 +0000 UTC m=+0.388032779 container init 1184bc13bc40423f8cd8b88a30e37d23ac5f8eefbe93eb6a685bcc42fb8f6d5a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, version=8.5, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, name=ubi8, release=236.1648460182) 2024-04-22T21:40:08.806 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 podman[38702]: 2024-04-22 21:40:08.575432528 +0000 UTC m=+0.390894868 container start 1184bc13bc40423f8cd8b88a30e37d23ac5f8eefbe93eb6a685bcc42fb8f6d5a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, description=Ceph Grafana Container, distribution-scope=public, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-22T21:40:08.806 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-22T21:40:08.807 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="App mode production" logger=settings 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-22T21:40:08.808 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-22T21:40:08.809 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-22T21:40:08.809 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-22T21:40:08.809 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-22T21:40:08.809 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-22T21:40:08.809 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 bash[38702]: 1184bc13bc40423f8cd8b88a30e37d23ac5f8eefbe93eb6a685bcc42fb8f6d5a 2024-04-22T21:40:08.810 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-22T21:40:08.810 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 systemd[1]: Started Ceph grafana.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:40:08.810 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-22T21:40:08.810 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-22T21:40:08.810 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-22T21:40:08.810 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-22T21:40:08.810 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-22T21:40:08.810 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-22T21:40:08.811 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-22T21:40:08.812 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-22T21:40:08.813 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-22T21:40:08.814 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-22T21:40:08.815 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-22T21:40:08.816 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-22T21:40:08.817 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-22T21:40:08.817 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-22T21:40:08.817 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-22T21:40:08.817 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-22T21:40:08.817 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-22T21:40:08.817 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-22T21:40:08.817 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-22T21:40:08.817 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-22T21:40:09.057 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-22T21:40:09.058 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-22T21:40:09.058 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-22T21:40:09.058 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-22T21:40:09.058 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-22T21:40:09.058 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-22T21:40:09.058 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-22T21:40:09.058 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-22T21:40:09.058 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-22T21:40:09.058 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-22T21:40:09.059 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-22T21:40:09.060 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-22T21:40:09.061 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-22T21:40:09.061 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-22T21:40:09.061 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-22T21:40:09.061 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-22T21:40:09.061 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-22T21:40:09.061 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-22T21:40:09.061 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-22T21:40:09.061 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-22T21:40:09.063 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-22T21:40:09.063 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-22T21:40:09.063 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-22T21:40:09.064 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-22T21:40:09.065 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-22T21:40:09.066 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-22T21:40:09.067 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-22T21:40:09.068 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-22T21:40:09.069 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-22T21:40:09.070 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-22T21:40:09.071 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-22T21:40:09.072 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-22T21:40:09.073 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-22T21:40:09.074 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-22T21:40:09.075 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-22T21:40:09.076 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-22T21:40:09.076 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-22T21:40:09.076 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-22T21:40:09.076 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-22T21:40:09.076 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-22T21:40:09.076 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-22T21:40:09.076 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-22T21:40:09.076 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-22T21:40:09.076 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-22T21:40:09.312 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-22T21:40:09.313 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-22T21:40:09.314 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-22T21:40:09.314 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-22T21:40:09.314 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-22T21:40:09.314 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-22T21:40:09.315 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-22T21:40:09.316 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-22T21:40:09.317 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-22T21:40:09.318 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=613.053091ms 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-22T21:40:09.319 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-22T21:40:09.320 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-22T21:40:09.688 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-22T21:40:09.689 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:40:09+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-22T21:40:10.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-mon[22698]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-22T21:40:10.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:10.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:10.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:09 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:10.045 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:09 smithi165 ceph-mon[26549]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-22T21:40:10.045 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:09 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:10.045 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:09 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:10.045 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:09 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:10.045 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:09 smithi165 ceph-mon[21833]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 1 op/s 2024-04-22T21:40:10.045 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:09 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:10.046 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:09 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:10.046 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:09 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:11.962 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:11 smithi165 ceph-mon[26549]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 1.9 KiB/s wr, 3 op/s 2024-04-22T21:40:11.962 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:11 smithi165 ceph-mon[26549]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T21:40:11.962 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:11 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-22T21:40:11.962 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:11 smithi165 ceph-mon[21833]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 1.9 KiB/s wr, 3 op/s 2024-04-22T21:40:11.962 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:11 smithi165 ceph-mon[21833]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T21:40:11.962 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:11 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-22T21:40:12.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:11 smithi188 ceph-mon[22698]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 1.9 KiB/s wr, 3 op/s 2024-04-22T21:40:12.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:11 smithi188 ceph-mon[22698]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T21:40:12.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:11 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-22T21:40:12.989 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-22T21:40:12.989 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[21833]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T21:40:12.989 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:12.989 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:12.990 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:12.990 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-22T21:40:12.990 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[26549]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T21:40:12.990 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:12.990 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:12.990 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:12 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:13.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:12 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-22T21:40:13.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:12 smithi188 ceph-mon[22698]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T21:40:13.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:12 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:13.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:12 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:13.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:12 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:14.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:13 smithi188 ceph-mon[22698]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-22T21:40:14.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:13 smithi188 ceph-mon[22698]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-22T21:40:14.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:13 smithi188 ceph-mon[22698]: Reconfiguring daemon alertmanager.a on smithi165 2024-04-22T21:40:14.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:13 smithi188 ceph-mon[22698]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T21:40:14.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:13 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-22T21:40:14.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:13 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:14.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:13 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-22T21:40:14.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:13 smithi188 ceph-mon[22698]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T21:40:14.057 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:13] "GET /metrics HTTP/1.1" 200 192901 "" "Prometheus/2.33.4" 2024-04-22T21:40:14.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[21833]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-22T21:40:14.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[21833]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[21833]: Reconfiguring daemon alertmanager.a on smithi165 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[21833]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[21833]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[26549]: pgmap v55: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[26549]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[26549]: Reconfiguring daemon alertmanager.a on smithi165 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[26549]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-22T21:40:14.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:14.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/110554945' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-22T21:40:14.059 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:13 smithi165 ceph-mon[26549]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T21:40:16.006 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:15 smithi165 ceph-mon[21833]: pgmap v58: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:16.006 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:15 smithi165 ceph-mon[21833]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T21:40:16.006 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:15 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3195437778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-22T21:40:16.006 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:15 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-22T21:40:16.006 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:15 smithi165 ceph-mon[26549]: pgmap v58: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:16.006 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:15 smithi165 ceph-mon[26549]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T21:40:16.007 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:15 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3195437778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-22T21:40:16.007 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:15 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-22T21:40:16.007 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:15 smithi165 bash[46394]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-22T21:40:16.007 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[45048]: level=info ts=2024-04-22T21:40:16.024Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-22T21:40:16.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:15 smithi188 ceph-mon[22698]: pgmap v58: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:16.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:15 smithi188 ceph-mon[22698]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T21:40:16.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:15 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3195437778' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-22T21:40:16.008 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:15 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-22T21:40:16.008 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:40:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:40:16.307 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 podman[46405]: 2024-04-22 21:40:16.055448189 +0000 UTC m=+0.133769422 container died a7b4bd7c96586b9e28ed21746b6c691e5434e9d2f64128ef12126d0ee8e9bcb2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:16.690 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 podman[46405]: 2024-04-22 21:40:16.4416005 +0000 UTC m=+0.519921739 container cleanup a7b4bd7c96586b9e28ed21746b6c691e5434e9d2f64128ef12126d0ee8e9bcb2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:16.690 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 bash[46405]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a 2024-04-22T21:40:16.690 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 bash[46435]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-22T21:40:17.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:16 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-22T21:40:17.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:16 smithi188 ceph-mon[22698]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T21:40:17.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:16 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3195437778' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-22T21:40:17.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:16 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-22T21:40:17.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:16 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-22T21:40:17.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:16 smithi188 ceph-mon[22698]: osdmap e70: 8 total, 8 up, 8 in 2024-04-22T21:40:17.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:16 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3195437778' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-22T21:40:17.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:16 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-22T21:40:17.016 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 podman[46419]: 2024-04-22 21:40:16.709508956 +0000 UTC m=+0.651609843 container remove a7b4bd7c96586b9e28ed21746b6c691e5434e9d2f64128ef12126d0ee8e9bcb2 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:17.016 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 podman[46419]: 2024-04-22 21:40:16.810435007 +0000 UTC m=+0.752535849 volume remove 7872b3554762b4985135f6d7c5586be97d3cd59870ce8571c32f856b109b1959 2024-04-22T21:40:17.017 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-22T21:40:17.017 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 systemd[1]: Stopped Ceph alertmanager.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:40:17.017 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:16 smithi165 systemd[1]: Starting Ceph alertmanager.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:40:17.017 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-22T21:40:17.017 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[21833]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T21:40:17.017 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3195437778' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-22T21:40:17.017 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-22T21:40:17.017 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-22T21:40:17.017 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[21833]: osdmap e70: 8 total, 8 up, 8 in 2024-04-22T21:40:17.017 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3195437778' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-22T21:40:17.017 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-22T21:40:17.018 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-22T21:40:17.018 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[26549]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T21:40:17.018 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3195437778' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-22T21:40:17.018 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-22T21:40:17.018 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-22T21:40:17.018 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[26549]: osdmap e70: 8 total, 8 up, 8 in 2024-04-22T21:40:17.018 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3195437778' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-22T21:40:17.018 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:16 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-22T21:40:17.294 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 podman[46496]: 2024-04-22 21:40:17.036190926 +0000 UTC m=+0.111403028 volume create 74e8bfbca2e1aafbec07d9740dd6ef91164c94a168d2892a36d1402b7b6ee5df 2024-04-22T21:40:17.294 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 podman[46496]: 2024-04-22 21:40:16.942648668 +0000 UTC m=+0.017860790 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-22T21:40:17.294 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 podman[46496]: 2024-04-22 21:40:17.120072349 +0000 UTC m=+0.195284452 container create 5f88b3382f5836e2bfa79ed9d9a8aa07c922014885b57ec2616e2f42badbe9a7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:17.745 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 podman[46496]: 2024-04-22 21:40:17.423343298 +0000 UTC m=+0.498555402 container init 5f88b3382f5836e2bfa79ed9d9a8aa07c922014885b57ec2616e2f42badbe9a7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:17.745 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 podman[46496]: 2024-04-22 21:40:17.426074191 +0000 UTC m=+0.501286295 container start 5f88b3382f5836e2bfa79ed9d9a8aa07c922014885b57ec2616e2f42badbe9a7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:17.745 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:17.436Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-22T21:40:17.745 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:17.436Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-22T21:40:17.745 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:17.437Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.165 port=9094 2024-04-22T21:40:17.745 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:17.438Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-22T21:40:17.746 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:17.463Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-22T21:40:17.746 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:17.463Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-22T21:40:17.746 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:17.465Z caller=main.go:518 msg=Listening address=:9093 2024-04-22T21:40:17.746 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:17.466Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-22T21:40:17.746 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 bash[46496]: 5f88b3382f5836e2bfa79ed9d9a8aa07c922014885b57ec2616e2f42badbe9a7 2024-04-22T21:40:17.746 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:17 smithi165 systemd[1]: Started Ceph alertmanager.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: "id": "9825d288-dcc5-4183-bfda-935d85806559", 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: "epoch": 1, 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: "predecessor_uuid": "fa325460-e75f-4ffa-865f-6b75c84fd264", 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: "sync_status": [], 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: "period_map": { 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: "id": "9825d288-dcc5-4183-bfda-935d85806559", 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: "zonegroups": [ 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: "id": "4a2dd0f3-a400-42cb-a3a5-2b9589a31b4b", 2024-04-22T21:40:17.787 INFO:teuthology.orchestra.run.smithi165.stdout: "name": "default", 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "api_name": "default", 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "is_master": "true", 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "endpoints": [], 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "hostnames": [], 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "hostnames_s3website": [], 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "master_zone": "32bf1395-a8ce-4d2c-bb1b-b6b6f5c218ff", 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "zones": [ 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "id": "32bf1395-a8ce-4d2c-bb1b-b6b6f5c218ff", 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "name": "z", 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "endpoints": [], 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "log_meta": "false", 2024-04-22T21:40:17.788 INFO:teuthology.orchestra.run.smithi165.stdout: "log_data": "false", 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: "bucket_index_max_shards": 11, 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: "read_only": "false", 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: "tier_type": "", 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: "sync_from_all": "true", 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: "sync_from": [], 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: "redirect_zone": "" 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: "placement_targets": [ 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-04-22T21:40:17.789 INFO:teuthology.orchestra.run.smithi165.stdout: "name": "default-placement", 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: "tags": [], 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: "storage_classes": [ 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: "STANDARD" 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: "default_placement": "default-placement", 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: "realm_id": "d2111914-a014-4462-a1c8-405f0fb86365", 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: "sync_policy": { 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: "groups": [] 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: ], 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: "short_zone_ids": [ 2024-04-22T21:40:17.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "key": "32bf1395-a8ce-4d2c-bb1b-b6b6f5c218ff", 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 2160493308 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "master_zonegroup": "4a2dd0f3-a400-42cb-a3a5-2b9589a31b4b", 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "master_zone": "32bf1395-a8ce-4d2c-bb1b-b6b6f5c218ff", 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "period_config": { 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "bucket_quota": { 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "enabled": false, 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "check_on_raw": false, 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "max_size": -1, 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "max_size_kb": 0, 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "max_objects": -1 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "user_quota": { 2024-04-22T21:40:17.791 INFO:teuthology.orchestra.run.smithi165.stdout: "enabled": false, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "check_on_raw": false, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_size": -1, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_size_kb": 0, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_objects": -1 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "user_ratelimit": { 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_read_ops": 0, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_write_ops": 0, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_read_bytes": 0, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_write_bytes": 0, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "enabled": false 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "bucket_ratelimit": { 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_read_ops": 0, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_write_ops": 0, 2024-04-22T21:40:17.792 INFO:teuthology.orchestra.run.smithi165.stdout: "max_read_bytes": 0, 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "max_write_bytes": 0, 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "enabled": false 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "anonymous_ratelimit": { 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "max_read_ops": 0, 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "max_write_ops": 0, 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "max_read_bytes": 0, 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "max_write_bytes": 0, 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "enabled": false 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "realm_id": "d2111914-a014-4462-a1c8-405f0fb86365", 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "realm_name": "r", 2024-04-22T21:40:17.793 INFO:teuthology.orchestra.run.smithi165.stdout: "realm_epoch": 2 2024-04-22T21:40:17.794 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-04-22T21:40:17.887 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:17 smithi188 ceph-mon[22698]: pgmap v61: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T21:40:17.888 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:17 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:17.888 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:17 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-22T21:40:17.888 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:17 smithi188 ceph-mon[22698]: osdmap e71: 8 total, 8 up, 8 in 2024-04-22T21:40:18.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-mon[21833]: pgmap v61: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T21:40:18.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:18.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-22T21:40:18.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-mon[21833]: osdmap e71: 8 total, 8 up, 8 in 2024-04-22T21:40:18.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-mon[26549]: pgmap v61: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T21:40:18.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:18.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-22T21:40:18.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:17 smithi165 ceph-mon[26549]: osdmap e71: 8 total, 8 up, 8 in 2024-04-22T21:40:18.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-22T21:40:18.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:18 smithi165 ceph-mon[21833]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-22T21:40:18.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:18 smithi165 ceph-mon[21833]: Reconfiguring daemon prometheus.a on smithi188 2024-04-22T21:40:18.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:18 smithi165 ceph-mon[26549]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-22T21:40:18.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:18 smithi165 ceph-mon[26549]: Reconfiguring daemon prometheus.a on smithi188 2024-04-22T21:40:19.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:18 smithi188 ceph-mon[22698]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-22T21:40:19.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:18 smithi188 ceph-mon[22698]: Reconfiguring daemon prometheus.a on smithi188 2024-04-22T21:40:19.751 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:19 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:19.439Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000381096s 2024-04-22T21:40:20.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:19 smithi188 ceph-mon[22698]: pgmap v64: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T21:40:20.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:19 smithi165 ceph-mon[21833]: pgmap v64: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T21:40:20.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:19 smithi165 ceph-mon[26549]: pgmap v64: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-22T21:40:20.493 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled rgw.foo update... 2024-04-22T21:40:21.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch apply rgw smpl' 2024-04-22T21:40:21.723 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:21 smithi165 ceph-mon[21833]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-22T21:40:21.723 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:21 smithi165 ceph-mon[21833]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:40:21.723 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:21 smithi165 ceph-mon[21833]: Saving service rgw.foo spec with placement count:2 2024-04-22T21:40:21.723 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:21 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:21.724 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:21 smithi165 ceph-mon[26549]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-22T21:40:21.724 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:21 smithi165 ceph-mon[26549]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:40:21.724 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:21 smithi165 ceph-mon[26549]: Saving service rgw.foo spec with placement count:2 2024-04-22T21:40:21.724 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:21 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:21.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:21 smithi188 ceph-mon[22698]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-22T21:40:21.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:21 smithi188 ceph-mon[22698]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:40:21.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:21 smithi188 ceph-mon[22698]: Saving service rgw.foo spec with placement count:2 2024-04-22T21:40:21.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:21 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:23.030 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled rgw.smpl update... 2024-04-22T21:40:23.757 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 systemd[1]: Stopping Ceph prometheus.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:40:23.757 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 bash[39387]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-22T21:40:23.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph osd pool create foo' 2024-04-22T21:40:24.060 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-mon[22698]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-22T21:40:24.061 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-mon[22698]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:40:24.061 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-mon[22698]: Saving service rgw.smpl spec with placement count:2 2024-04-22T21:40:24.061 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:24.061 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.803Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-22T21:40:24.061 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-22T21:40:24.061 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-22T21:40:24.061 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-22T21:40:24.061 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-22T21:40:24.061 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-22T21:40:24.061 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-22T21:40:24.062 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-22T21:40:24.062 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-22T21:40:24.062 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-22T21:40:24.062 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-22T21:40:24.062 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[37717]: ts=2024-04-22T21:40:23.804Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-22T21:40:24.062 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:23 smithi188 podman[39399]: 2024-04-22 21:40:23.81416087 +0000 UTC m=+0.124859690 container died 70bc8fb45a33ef4058c19019076cad8567e9d35c2ef7b120fe68bce9765136e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:24.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:24 smithi165 ceph-mon[21833]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-22T21:40:24.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:24 smithi165 ceph-mon[21833]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:40:24.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:24 smithi165 ceph-mon[21833]: Saving service rgw.smpl spec with placement count:2 2024-04-22T21:40:24.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:24 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:24.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:24 smithi165 ceph-mon[26549]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-22T21:40:24.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:24 smithi165 ceph-mon[26549]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:40:24.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:24 smithi165 ceph-mon[26549]: Saving service rgw.smpl spec with placement count:2 2024-04-22T21:40:24.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:24 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:24.436 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 podman[39399]: 2024-04-22 21:40:24.187322946 +0000 UTC m=+0.498021765 container cleanup 70bc8fb45a33ef4058c19019076cad8567e9d35c2ef7b120fe68bce9765136e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:24.436 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 bash[39399]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a 2024-04-22T21:40:24.436 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 bash[39425]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-22T21:40:24.703 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 podman[39413]: 2024-04-22 21:40:24.455347285 +0000 UTC m=+0.643156699 container remove 70bc8fb45a33ef4058c19019076cad8567e9d35c2ef7b120fe68bce9765136e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:24.703 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-22T21:40:24.703 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 systemd[1]: Stopped Ceph prometheus.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:40:24.703 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 systemd[1]: Starting Ceph prometheus.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:40:24.703 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 podman[39493]: 2024-04-22 21:40:24.589979401 +0000 UTC m=+0.018810427 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-22T21:40:24.704 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 podman[39493]: 2024-04-22 21:40:24.722793256 +0000 UTC m=+0.151624280 container create b1b3a948cedf3217709263558fa1c118ccdd2501898c6a5c30cdb33c810ba5bf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:24.956 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 podman[39493]: 2024-04-22 21:40:24.950238424 +0000 UTC m=+0.379069449 container init b1b3a948cedf3217709263558fa1c118ccdd2501898c6a5c30cdb33c810ba5bf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:24.956 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 podman[39493]: 2024-04-22 21:40:24.952709231 +0000 UTC m=+0.381540257 container start b1b3a948cedf3217709263558fa1c118ccdd2501898c6a5c30cdb33c810ba5bf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:40:25.257 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.975Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-22T21:40:25.257 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.975Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-22T21:40:25.257 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.975Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-22T21:40:25.257 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.975Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi188 (none))" 2024-04-22T21:40:25.257 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.975Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.975Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.977Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.977Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.978Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.980Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.981Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.536µs 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:24 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:24.981Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 bash[39493]: b1b3a948cedf3217709263558fa1c118ccdd2501898c6a5c30cdb33c810ba5bf 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 systemd[1]: Started Ceph prometheus.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:25.188Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:25.188Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:25.188Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.111µs wal_replay_duration=207.376204ms total_replay_duration=207.41483ms 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:25.190Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-22T21:40:25.258 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:25.190Z caller=main.go:947 level=info msg="TSDB started" 2024-04-22T21:40:25.259 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:25.190Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-22T21:40:25.259 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:25.206Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.34732ms db_storage=728ns remote_storage=1.266µs web_handler=430ns query_engine=684ns scrape=427.137µs scrape_sd=24.679µs notify=27.217µs notify_sd=9.422µs rules=15.621285ms 2024-04-22T21:40:25.259 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:40:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:40:25.206Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-22T21:40:26.105 INFO:teuthology.orchestra.run.smithi165.stderr:pool 'foo' created 2024-04-22T21:40:26.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.165:9093"}]: dispatch 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.165:9093"}]: dispatch 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.188:9095"}]: dispatch 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.188:9095"}]: dispatch 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:40:26.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.188:3000"}]: dispatch 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.188:3000"}]: dispatch 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.165:9093"}]: dispatch 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.165:9093"}]: dispatch 2024-04-22T21:40:26.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.188:9095"}]: dispatch 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.188:9095"}]: dispatch 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.188:3000"}]: dispatch 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.188:3000"}]: dispatch 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:26.310 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:26 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:26.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-04-22T21:40:26.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.165:9093"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.165:9093"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.188:9095"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.188:9095"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.188:3000"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.188:3000"}]: dispatch 2024-04-22T21:40:26.508 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:26.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:26.509 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:26 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:27.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'rbd pool init foo' 2024-04-22T21:40:27.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:27 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2314450704' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T21:40:27.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:27 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T21:40:27.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:27 smithi165 ceph-mon[21833]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-22T21:40:27.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:27 smithi165 ceph-mon[21833]: osdmap e72: 8 total, 8 up, 8 in 2024-04-22T21:40:27.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:27 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/2314450704' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T21:40:27.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:27 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T21:40:27.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:27 smithi165 ceph-mon[26549]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-22T21:40:27.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:27 smithi165 ceph-mon[26549]: osdmap e72: 8 total, 8 up, 8 in 2024-04-22T21:40:27.416 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:27 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/2314450704' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T21:40:27.416 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:27 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T21:40:27.416 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:27 smithi188 ceph-mon[22698]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-22T21:40:27.416 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:27 smithi188 ceph-mon[22698]: osdmap e72: 8 total, 8 up, 8 in 2024-04-22T21:40:27.751 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:40:27 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:40:27.440Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002145993s 2024-04-22T21:40:28.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:28 smithi188 ceph-mon[22698]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 829 B/s wr, 2 op/s 2024-04-22T21:40:28.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:28 smithi188 ceph-mon[22698]: osdmap e73: 8 total, 8 up, 8 in 2024-04-22T21:40:28.460 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:28 smithi165 ceph-mon[21833]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 829 B/s wr, 2 op/s 2024-04-22T21:40:28.460 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:28 smithi165 ceph-mon[21833]: osdmap e73: 8 total, 8 up, 8 in 2024-04-22T21:40:28.460 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:28 smithi165 ceph-mon[26549]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 829 B/s wr, 2 op/s 2024-04-22T21:40:28.460 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:28 smithi165 ceph-mon[26549]: osdmap e73: 8 total, 8 up, 8 in 2024-04-22T21:40:29.491 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:29 smithi165 ceph-mon[26549]: osdmap e74: 8 total, 8 up, 8 in 2024-04-22T21:40:29.491 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:29 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:29.491 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:29 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:29.492 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:29 smithi165 ceph-mon[21833]: osdmap e74: 8 total, 8 up, 8 in 2024-04-22T21:40:29.492 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:29 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:29.492 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:29 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:29.506 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:29 smithi188 ceph-mon[22698]: osdmap e74: 8 total, 8 up, 8 in 2024-04-22T21:40:29.506 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:29 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:29.506 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:29 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:30.367 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:30 smithi165 ceph-mon[21833]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:30.367 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:30 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3887127082' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-22T21:40:30.367 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:30 smithi165 ceph-mon[26549]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:30.367 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:30 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3887127082' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-22T21:40:30.449 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:30 smithi188 ceph-mon[22698]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:30.449 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:30 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3887127082' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3887127082' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: osdmap e75: 8 total, 8 up, 8 in 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi165.pbstvw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi165.pbstvw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:31 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:31.550 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3887127082' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-22T21:40:31.550 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: osdmap e75: 8 total, 8 up, 8 in 2024-04-22T21:40:31.550 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi165.pbstvw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi165.pbstvw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3887127082' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-22T21:40:31.551 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: osdmap e75: 8 total, 8 up, 8 in 2024-04-22T21:40:31.552 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.552 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.552 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.552 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.552 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.552 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi165.pbstvw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:31.552 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi165.pbstvw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:31.552 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:31.552 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:31 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:32.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:32 smithi165 ceph-mon[21833]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:32.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:32 smithi165 ceph-mon[21833]: Saving service rgw.foo spec with placement count:2 2024-04-22T21:40:32.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:32 smithi165 ceph-mon[21833]: Deploying daemon rgw.foo.smithi165.pbstvw on smithi165 2024-04-22T21:40:32.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:32 smithi165 ceph-mon[21833]: osdmap e76: 8 total, 8 up, 8 in 2024-04-22T21:40:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:32 smithi165 ceph-mon[26549]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:32 smithi165 ceph-mon[26549]: Saving service rgw.foo spec with placement count:2 2024-04-22T21:40:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:32 smithi165 ceph-mon[26549]: Deploying daemon rgw.foo.smithi165.pbstvw on smithi165 2024-04-22T21:40:32.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:32 smithi165 ceph-mon[26549]: osdmap e76: 8 total, 8 up, 8 in 2024-04-22T21:40:32.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:32 smithi188 ceph-mon[22698]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:32.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:32 smithi188 ceph-mon[22698]: Saving service rgw.foo spec with placement count:2 2024-04-22T21:40:32.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:32 smithi188 ceph-mon[22698]: Deploying daemon rgw.foo.smithi165.pbstvw on smithi165 2024-04-22T21:40:32.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:32 smithi188 ceph-mon[22698]: osdmap e76: 8 total, 8 up, 8 in 2024-04-22T21:40:32.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-22T21:40:33.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:33 smithi165 ceph-mon[21833]: osdmap e77: 8 total, 8 up, 8 in 2024-04-22T21:40:33.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:33 smithi165 ceph-mon[26549]: osdmap e77: 8 total, 8 up, 8 in 2024-04-22T21:40:33.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:33 smithi188 ceph-mon[22698]: osdmap e77: 8 total, 8 up, 8 in 2024-04-22T21:40:34.206 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:34 smithi165 ceph-mon[21833]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:34.206 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:34 smithi165 ceph-mon[26549]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:34.207 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:40:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:33] "GET /metrics HTTP/1.1" 200 196805 "" "Prometheus/2.33.4" 2024-04-22T21:40:34.506 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:34 smithi188 ceph-mon[22698]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:40:35.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:35 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:35.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:35 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi188.wgqixq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:35.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:35 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi188.wgqixq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:35.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:35 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:35.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:35 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:35.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:35 smithi188 ceph-mon[22698]: Deploying daemon rgw.foo.smithi188.wgqixq on smithi188 2024-04-22T21:40:35.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:35 smithi188 ceph-mon[22698]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 511 B/s wr, 16 op/s 2024-04-22T21:40:35.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:35.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi188.wgqixq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:35.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi188.wgqixq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:35.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:35.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:35.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[26549]: Deploying daemon rgw.foo.smithi188.wgqixq on smithi188 2024-04-22T21:40:35.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[26549]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 511 B/s wr, 16 op/s 2024-04-22T21:40:35.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:35.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi188.wgqixq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:35.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi188.wgqixq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:35.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:35.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:35.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[21833]: Deploying daemon rgw.foo.smithi188.wgqixq on smithi188 2024-04-22T21:40:35.309 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:35 smithi165 ceph-mon[21833]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 511 B/s wr, 16 op/s 2024-04-22T21:40:35.430 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled iscsi.foo update... 2024-04-22T21:40:36.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'sleep 120' 2024-04-22T21:40:36.230 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:40:35 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:40:36.506 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:36 smithi188 ceph-mon[22698]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:40:36.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:36 smithi188 ceph-mon[22698]: Saving service iscsi.foo spec with placement count:1 2024-04-22T21:40:36.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:36 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:36.706 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:36 smithi165 ceph-mon[26549]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:40:36.707 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:36 smithi165 ceph-mon[26549]: Saving service iscsi.foo spec with placement count:1 2024-04-22T21:40:36.707 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:36 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:36.707 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:36 smithi165 ceph-mon[21833]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:40:36.707 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:36 smithi165 ceph-mon[21833]: Saving service iscsi.foo spec with placement count:1 2024-04-22T21:40:36.707 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:36 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:37.867 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 5.6 KiB/s wr, 131 op/s 2024-04-22T21:40:37.867 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: Saving service rgw.smpl spec with placement count:2 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi165.hjvsmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi165.hjvsmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: Deploying daemon rgw.smpl.smithi165.hjvsmr on smithi165 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[21833]: osdmap e78: 8 total, 8 up, 8 in 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 5.6 KiB/s wr, 131 op/s 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: Saving service rgw.smpl spec with placement count:2 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:37.868 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi165.hjvsmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:37.869 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi165.hjvsmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:37.869 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:37.869 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:37.869 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: Deploying daemon rgw.smpl.smithi165.hjvsmr on smithi165 2024-04-22T21:40:37.869 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:37 smithi165 ceph-mon[26549]: osdmap e78: 8 total, 8 up, 8 in 2024-04-22T21:40:38.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 5.6 KiB/s wr, 131 op/s 2024-04-22T21:40:38.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:38.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: Saving service rgw.smpl spec with placement count:2 2024-04-22T21:40:38.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:38.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi165.hjvsmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:38.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi165.hjvsmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:38.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:38.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:38.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: Deploying daemon rgw.smpl.smithi165.hjvsmr on smithi165 2024-04-22T21:40:38.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:37 smithi188 ceph-mon[22698]: osdmap e78: 8 total, 8 up, 8 in 2024-04-22T21:40:38.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-22T21:40:38.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-22T21:40:38.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:40:38.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:39.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-04-22T21:40:39.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-22T21:40:39.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-22T21:40:39.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-04-22T21:40:39.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:40:39.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:40:39.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:39.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[21833]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 4.8 KiB/s wr, 113 op/s 2024-04-22T21:40:39.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-04-22T21:40:39.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-22T21:40:39.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-22T21:40:39.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-04-22T21:40:39.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[21833]: osdmap e79: 8 total, 8 up, 8 in 2024-04-22T21:40:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[26549]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 4.8 KiB/s wr, 113 op/s 2024-04-22T21:40:39.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-04-22T21:40:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-22T21:40:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-22T21:40:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-04-22T21:40:39.809 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:39 smithi165 ceph-mon[26549]: osdmap e79: 8 total, 8 up, 8 in 2024-04-22T21:40:40.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:39 smithi188 ceph-mon[22698]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 4.8 KiB/s wr, 113 op/s 2024-04-22T21:40:40.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-04-22T21:40:40.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-22T21:40:40.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-22T21:40:40.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-04-22T21:40:40.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:39 smithi188 ceph-mon[22698]: osdmap e79: 8 total, 8 up, 8 in 2024-04-22T21:40:40.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:40 smithi188 ceph-mon[22698]: osdmap e80: 8 total, 8 up, 8 in 2024-04-22T21:40:40.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:40 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:40.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:40 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi188.ryxylz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:40.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:40 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi188.ryxylz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:40.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:40 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:40.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:40 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:40.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[26549]: osdmap e80: 8 total, 8 up, 8 in 2024-04-22T21:40:40.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:40.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi188.ryxylz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:40.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi188.ryxylz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:40.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:40.808 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:40.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[21833]: osdmap e80: 8 total, 8 up, 8 in 2024-04-22T21:40:40.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:40.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi188.ryxylz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T21:40:40.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi188.ryxylz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T21:40:40.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:40.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:40 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:41.996 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:41 smithi188 ceph-mon[22698]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 5.5 KiB/s wr, 337 op/s 2024-04-22T21:40:41.996 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:41 smithi188 ceph-mon[22698]: Deploying daemon rgw.smpl.smithi188.ryxylz on smithi188 2024-04-22T21:40:42.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:41 smithi165 ceph-mon[21833]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 5.5 KiB/s wr, 337 op/s 2024-04-22T21:40:42.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:41 smithi165 ceph-mon[21833]: Deploying daemon rgw.smpl.smithi188.ryxylz on smithi188 2024-04-22T21:40:42.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:41 smithi165 ceph-mon[26549]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 5.5 KiB/s wr, 337 op/s 2024-04-22T21:40:42.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:41 smithi165 ceph-mon[26549]: Deploying daemon rgw.smpl.smithi188.ryxylz on smithi188 2024-04-22T21:40:44.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:40:43 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:43] "GET /metrics HTTP/1.1" 200 206392 "" "Prometheus/2.33.4" 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[21833]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 170 B/s wr, 216 op/s 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[26549]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 170 B/s wr, 216 op/s 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:44.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:44 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:44.336 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:44 smithi188 ceph-mon[22698]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 170 B/s wr, 216 op/s 2024-04-22T21:40:44.336 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:44 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:44.336 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:44 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:44.336 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:44 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:44.336 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:44 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:46.250 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:40:45 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:40:46.250 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:46 smithi188 ceph-mon[22698]: pgmap v86: 161 pgs: 1 peering, 160 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 195 KiB/s rd, 437 B/s wr, 318 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:46.250 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:46 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:46.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:46 smithi165 ceph-mon[21833]: pgmap v86: 161 pgs: 1 peering, 160 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 195 KiB/s rd, 437 B/s wr, 318 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:46.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:46 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:46.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:46 smithi165 ceph-mon[26549]: pgmap v86: 161 pgs: 1 peering, 160 active+clean; 584 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 195 KiB/s rd, 437 B/s wr, 318 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:46.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:46 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 258 KiB/s rd, 383 B/s wr, 413 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: Checking dashboard <-> RGW credentials 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: Checking pool "foo" exists for service iscsi.foo 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi165.ceupac", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi165.ceupac", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:48.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:47 smithi188 ceph-mon[22698]: Deploying daemon iscsi.foo.smithi165.ceupac on smithi165 2024-04-22T21:40:48.049 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 258 KiB/s rd, 383 B/s wr, 413 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:48.049 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.049 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: Checking dashboard <-> RGW credentials 2024-04-22T21:40:48.049 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.049 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: Checking pool "foo" exists for service iscsi.foo 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi165.ceupac", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi165.ceupac", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[21833]: Deploying daemon iscsi.foo.smithi165.ceupac on smithi165 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 258 KiB/s rd, 383 B/s wr, 413 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: Checking dashboard <-> RGW credentials 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.050 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.051 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:48.051 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: Checking pool "foo" exists for service iscsi.foo 2024-04-22T21:40:48.051 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi165.ceupac", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-22T21:40:48.051 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi165.ceupac", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-22T21:40:48.051 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:48.051 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:47 smithi165 ceph-mon[26549]: Deploying daemon iscsi.foo.smithi165.ceupac on smithi165 2024-04-22T21:40:49.802 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:49 smithi165 ceph-mon[21833]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 212 KiB/s rd, 315 B/s wr, 339 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:49.802 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:49 smithi165 ceph-mon[21833]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-22T21:40:49.803 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:49 smithi165 ceph-mon[26549]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 212 KiB/s rd, 315 B/s wr, 339 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:49.803 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:49 smithi165 ceph-mon[26549]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-22T21:40:50.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:49 smithi188 ceph-mon[22698]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 212 KiB/s rd, 315 B/s wr, 339 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:49 smithi188 ceph-mon[22698]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-22T21:40:51.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:51.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:51.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:51.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3243374090' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T21:40:51.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/288085368' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/229165957"}]: dispatch 2024-04-22T21:40:51.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:51.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:51.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:51.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3243374090' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T21:40:51.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:51 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/288085368' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/229165957"}]: dispatch 2024-04-22T21:40:51.506 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:51 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:51.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:51 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:51.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:51 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:51.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:51 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3243374090' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T21:40:51.507 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:51 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/288085368' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/229165957"}]: dispatch 2024-04-22T21:40:52.382 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:52 smithi188 ceph-mon[22698]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 381 B/s wr, 191 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:52.382 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:52 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/288085368' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/229165957"}]': finished 2024-04-22T21:40:52.385 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:52 smithi188 ceph-mon[22698]: osdmap e81: 8 total, 8 up, 8 in 2024-04-22T21:40:52.385 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:52 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3277298614' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/438332672"}]: dispatch 2024-04-22T21:40:52.385 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:52 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/438332672"}]: dispatch 2024-04-22T21:40:52.414 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[26549]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 381 B/s wr, 191 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:52.414 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/288085368' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/229165957"}]': finished 2024-04-22T21:40:52.414 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[26549]: osdmap e81: 8 total, 8 up, 8 in 2024-04-22T21:40:52.414 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3277298614' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/438332672"}]: dispatch 2024-04-22T21:40:52.414 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/438332672"}]: dispatch 2024-04-22T21:40:52.415 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[21833]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 381 B/s wr, 191 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T21:40:52.415 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/288085368' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/229165957"}]': finished 2024-04-22T21:40:52.415 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[21833]: osdmap e81: 8 total, 8 up, 8 in 2024-04-22T21:40:52.415 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3277298614' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/438332672"}]: dispatch 2024-04-22T21:40:52.415 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:52 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/438332672"}]: dispatch 2024-04-22T21:40:53.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:53 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/438332672"}]': finished 2024-04-22T21:40:53.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:53 smithi165 ceph-mon[21833]: osdmap e82: 8 total, 8 up, 8 in 2024-04-22T21:40:53.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:53 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3529821474' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/2127003174"}]: dispatch 2024-04-22T21:40:53.307 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:53 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/2127003174"}]: dispatch 2024-04-22T21:40:53.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:53 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/438332672"}]': finished 2024-04-22T21:40:53.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:53 smithi165 ceph-mon[26549]: osdmap e82: 8 total, 8 up, 8 in 2024-04-22T21:40:53.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:53 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3529821474' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/2127003174"}]: dispatch 2024-04-22T21:40:53.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:53 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/2127003174"}]: dispatch 2024-04-22T21:40:53.337 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:53 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/438332672"}]': finished 2024-04-22T21:40:53.337 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:53 smithi188 ceph-mon[22698]: osdmap e82: 8 total, 8 up, 8 in 2024-04-22T21:40:53.337 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:53 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3529821474' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/2127003174"}]: dispatch 2024-04-22T21:40:53.337 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:53 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/2127003174"}]: dispatch 2024-04-22T21:40:54.246 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 255 B/s wr, 139 op/s 2024-04-22T21:40:54.246 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/2127003174"}]': finished 2024-04-22T21:40:54.246 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: osdmap e83: 8 total, 8 up, 8 in 2024-04-22T21:40:54.246 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.246 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.246 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.246 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.246 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi165"}]: dispatch 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/62811804' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/2127003174"}]: dispatch 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 255 B/s wr, 139 op/s 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/2127003174"}]': finished 2024-04-22T21:40:54.247 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: osdmap e83: 8 total, 8 up, 8 in 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi165"}]: dispatch 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:54 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/62811804' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/2127003174"}]: dispatch 2024-04-22T21:40:54.248 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:40:53 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:53] "GET /metrics HTTP/1.1" 200 214839 "" "Prometheus/2.33.4" 2024-04-22T21:40:54.348 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 255 B/s wr, 139 op/s 2024-04-22T21:40:54.348 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/2127003174"}]': finished 2024-04-22T21:40:54.348 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: osdmap e83: 8 total, 8 up, 8 in 2024-04-22T21:40:54.348 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.348 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.348 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.348 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.348 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T21:40:54.348 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T21:40:54.349 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.349 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi165"}]: dispatch 2024-04-22T21:40:54.349 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:54.349 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:40:54.349 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:40:54.349 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:54 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/62811804' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/2127003174"}]: dispatch 2024-04-22T21:40:55.327 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:55 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T21:40:55.327 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:55 smithi188 ceph-mon[22698]: Adding iSCSI gateway http://:@172.21.15.165:5000 to Dashboard 2024-04-22T21:40:55.327 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:55 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T21:40:55.327 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:55 smithi188 ceph-mon[22698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi165"}]: dispatch 2024-04-22T21:40:55.327 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:55 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/62811804' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/2127003174"}]': finished 2024-04-22T21:40:55.327 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:55 smithi188 ceph-mon[22698]: osdmap e84: 8 total, 8 up, 8 in 2024-04-22T21:40:55.327 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:55 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1669327194' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1056841091"}]: dispatch 2024-04-22T21:40:55.328 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:55 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1056841091"}]: dispatch 2024-04-22T21:40:55.381 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T21:40:55.382 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[21833]: Adding iSCSI gateway http://:@172.21.15.165:5000 to Dashboard 2024-04-22T21:40:55.382 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T21:40:55.382 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[21833]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi165"}]: dispatch 2024-04-22T21:40:55.382 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/62811804' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/2127003174"}]': finished 2024-04-22T21:40:55.382 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[21833]: osdmap e84: 8 total, 8 up, 8 in 2024-04-22T21:40:55.382 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1669327194' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1056841091"}]: dispatch 2024-04-22T21:40:55.382 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1056841091"}]: dispatch 2024-04-22T21:40:55.382 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T21:40:55.382 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[26549]: Adding iSCSI gateway http://:@172.21.15.165:5000 to Dashboard 2024-04-22T21:40:55.383 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T21:40:55.383 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[26549]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi165"}]: dispatch 2024-04-22T21:40:55.383 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/62811804' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/2127003174"}]': finished 2024-04-22T21:40:55.383 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[26549]: osdmap e84: 8 total, 8 up, 8 in 2024-04-22T21:40:55.383 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1669327194' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1056841091"}]: dispatch 2024-04-22T21:40:55.383 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:55 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1056841091"}]: dispatch 2024-04-22T21:40:56.257 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:40:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:40:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:40:56.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:56 smithi188 ceph-mon[22698]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 7.8 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T21:40:56.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:56 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1056841091"}]': finished 2024-04-22T21:40:56.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:56 smithi188 ceph-mon[22698]: osdmap e85: 8 total, 8 up, 8 in 2024-04-22T21:40:56.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:56 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/776572035' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2427714119"}]: dispatch 2024-04-22T21:40:56.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:56 smithi165 ceph-mon[26549]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 7.8 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T21:40:56.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:56 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1056841091"}]': finished 2024-04-22T21:40:56.307 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:56 smithi165 ceph-mon[26549]: osdmap e85: 8 total, 8 up, 8 in 2024-04-22T21:40:56.308 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:56 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/776572035' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2427714119"}]: dispatch 2024-04-22T21:40:56.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:56 smithi165 ceph-mon[21833]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 7.8 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-22T21:40:56.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:56 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1056841091"}]': finished 2024-04-22T21:40:56.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:56 smithi165 ceph-mon[21833]: osdmap e85: 8 total, 8 up, 8 in 2024-04-22T21:40:56.308 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:56 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/776572035' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2427714119"}]: dispatch 2024-04-22T21:40:58.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:57 smithi188 ceph-mon[22698]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 237 B/s wr, 3 op/s 2024-04-22T21:40:58.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:57 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/776572035' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2427714119"}]': finished 2024-04-22T21:40:58.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:57 smithi188 ceph-mon[22698]: osdmap e86: 8 total, 8 up, 8 in 2024-04-22T21:40:58.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:57 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3980687692' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2436844333"}]: dispatch 2024-04-22T21:40:58.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:57 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:58.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:57 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:58.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:57 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:58.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[21833]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 237 B/s wr, 3 op/s 2024-04-22T21:40:58.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/776572035' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2427714119"}]': finished 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[21833]: osdmap e86: 8 total, 8 up, 8 in 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3980687692' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2436844333"}]: dispatch 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[26549]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 237 B/s wr, 3 op/s 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/776572035' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2427714119"}]': finished 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[26549]: osdmap e86: 8 total, 8 up, 8 in 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3980687692' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2436844333"}]: dispatch 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:58.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:57 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:40:59.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:58 smithi188 ceph-mon[22698]: Checking dashboard <-> RGW credentials 2024-04-22T21:40:59.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:58 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3980687692' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2436844333"}]': finished 2024-04-22T21:40:59.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:58 smithi188 ceph-mon[22698]: osdmap e87: 8 total, 8 up, 8 in 2024-04-22T21:40:59.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:58 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1583740232' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/4281613920"}]: dispatch 2024-04-22T21:40:59.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:58 smithi165 ceph-mon[21833]: Checking dashboard <-> RGW credentials 2024-04-22T21:40:59.100 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:58 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3980687692' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2436844333"}]': finished 2024-04-22T21:40:59.101 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:58 smithi165 ceph-mon[21833]: osdmap e87: 8 total, 8 up, 8 in 2024-04-22T21:40:59.101 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:58 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1583740232' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/4281613920"}]: dispatch 2024-04-22T21:40:59.101 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:58 smithi165 ceph-mon[26549]: Checking dashboard <-> RGW credentials 2024-04-22T21:40:59.101 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:58 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3980687692' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/2436844333"}]': finished 2024-04-22T21:40:59.101 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:58 smithi165 ceph-mon[26549]: osdmap e87: 8 total, 8 up, 8 in 2024-04-22T21:40:59.101 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:58 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1583740232' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/4281613920"}]: dispatch 2024-04-22T21:41:00.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:59 smithi188 ceph-mon[22698]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:41:00.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:59 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1583740232' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/4281613920"}]': finished 2024-04-22T21:41:00.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:59 smithi188 ceph-mon[22698]: osdmap e88: 8 total, 8 up, 8 in 2024-04-22T21:41:00.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:40:59 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3058491238' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1312953247"}]: dispatch 2024-04-22T21:41:00.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:59 smithi165 ceph-mon[21833]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:41:00.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:59 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1583740232' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/4281613920"}]': finished 2024-04-22T21:41:00.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:59 smithi165 ceph-mon[21833]: osdmap e88: 8 total, 8 up, 8 in 2024-04-22T21:41:00.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:40:59 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3058491238' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1312953247"}]: dispatch 2024-04-22T21:41:00.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:59 smithi165 ceph-mon[26549]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-04-22T21:41:00.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:59 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1583740232' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/4281613920"}]': finished 2024-04-22T21:41:00.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:59 smithi165 ceph-mon[26549]: osdmap e88: 8 total, 8 up, 8 in 2024-04-22T21:41:00.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:40:59 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3058491238' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1312953247"}]: dispatch 2024-04-22T21:41:01.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:00 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3058491238' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1312953247"}]': finished 2024-04-22T21:41:01.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:00 smithi188 ceph-mon[22698]: osdmap e89: 8 total, 8 up, 8 in 2024-04-22T21:41:01.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:00 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/776904328' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3661503622"}]: dispatch 2024-04-22T21:41:01.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:00 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3661503622"}]: dispatch 2024-04-22T21:41:01.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:00 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3058491238' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1312953247"}]': finished 2024-04-22T21:41:01.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:00 smithi165 ceph-mon[21833]: osdmap e89: 8 total, 8 up, 8 in 2024-04-22T21:41:01.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:00 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/776904328' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3661503622"}]: dispatch 2024-04-22T21:41:01.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:00 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3661503622"}]: dispatch 2024-04-22T21:41:01.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:00 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3058491238' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1312953247"}]': finished 2024-04-22T21:41:01.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:00 smithi165 ceph-mon[26549]: osdmap e89: 8 total, 8 up, 8 in 2024-04-22T21:41:01.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:00 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/776904328' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3661503622"}]: dispatch 2024-04-22T21:41:01.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:00 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3661503622"}]: dispatch 2024-04-22T21:41:02.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:01 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:02.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:01 smithi188 ceph-mon[22698]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-22T21:41:02.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:01 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3661503622"}]': finished 2024-04-22T21:41:02.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:01 smithi188 ceph-mon[22698]: osdmap e90: 8 total, 8 up, 8 in 2024-04-22T21:41:02.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:01 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/2207858979' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3105631583"}]: dispatch 2024-04-22T21:41:02.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:01 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3105631583"}]: dispatch 2024-04-22T21:41:02.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:02.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[21833]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-22T21:41:02.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3661503622"}]': finished 2024-04-22T21:41:02.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[21833]: osdmap e90: 8 total, 8 up, 8 in 2024-04-22T21:41:02.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2207858979' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3105631583"}]: dispatch 2024-04-22T21:41:02.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3105631583"}]: dispatch 2024-04-22T21:41:02.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:02.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[26549]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-22T21:41:02.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3661503622"}]': finished 2024-04-22T21:41:02.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[26549]: osdmap e90: 8 total, 8 up, 8 in 2024-04-22T21:41:02.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/2207858979' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3105631583"}]: dispatch 2024-04-22T21:41:02.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:01 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3105631583"}]: dispatch 2024-04-22T21:41:03.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:02 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3105631583"}]': finished 2024-04-22T21:41:03.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:02 smithi188 ceph-mon[22698]: osdmap e91: 8 total, 8 up, 8 in 2024-04-22T21:41:03.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:02 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3038312403' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/438332672"}]: dispatch 2024-04-22T21:41:03.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:02 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3105631583"}]': finished 2024-04-22T21:41:03.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:02 smithi165 ceph-mon[21833]: osdmap e91: 8 total, 8 up, 8 in 2024-04-22T21:41:03.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:02 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3038312403' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/438332672"}]: dispatch 2024-04-22T21:41:03.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:02 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3105631583"}]': finished 2024-04-22T21:41:03.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:02 smithi165 ceph-mon[26549]: osdmap e91: 8 total, 8 up, 8 in 2024-04-22T21:41:03.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:02 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3038312403' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/438332672"}]: dispatch 2024-04-22T21:41:03.994 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[21833]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-22T21:41:03.994 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/3038312403' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/438332672"}]': finished 2024-04-22T21:41:03.994 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[21833]: osdmap e92: 8 total, 8 up, 8 in 2024-04-22T21:41:03.994 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/4039746690' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1483522820"}]: dispatch 2024-04-22T21:41:03.995 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1483522820"}]: dispatch 2024-04-22T21:41:03.995 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:03] "GET /metrics HTTP/1.1" 200 214839 "" "Prometheus/2.33.4" 2024-04-22T21:41:03.995 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[26549]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-22T21:41:03.995 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/3038312403' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/438332672"}]': finished 2024-04-22T21:41:03.995 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[26549]: osdmap e92: 8 total, 8 up, 8 in 2024-04-22T21:41:03.995 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/4039746690' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1483522820"}]: dispatch 2024-04-22T21:41:03.995 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:03 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1483522820"}]: dispatch 2024-04-22T21:41:04.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:03 smithi188 ceph-mon[22698]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-22T21:41:04.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:03 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/3038312403' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6800/438332672"}]': finished 2024-04-22T21:41:04.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:03 smithi188 ceph-mon[22698]: osdmap e92: 8 total, 8 up, 8 in 2024-04-22T21:41:04.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:03 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/4039746690' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1483522820"}]: dispatch 2024-04-22T21:41:04.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:03 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1483522820"}]: dispatch 2024-04-22T21:41:05.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:04 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1483522820"}]': finished 2024-04-22T21:41:05.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:04 smithi188 ceph-mon[22698]: osdmap e93: 8 total, 8 up, 8 in 2024-04-22T21:41:05.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:04 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1480794756' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/4281613920"}]: dispatch 2024-04-22T21:41:05.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:04 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1483522820"}]': finished 2024-04-22T21:41:05.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:04 smithi165 ceph-mon[21833]: osdmap e93: 8 total, 8 up, 8 in 2024-04-22T21:41:05.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:04 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1480794756' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/4281613920"}]: dispatch 2024-04-22T21:41:05.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:04 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/1483522820"}]': finished 2024-04-22T21:41:05.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:04 smithi165 ceph-mon[26549]: osdmap e93: 8 total, 8 up, 8 in 2024-04-22T21:41:05.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:04 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1480794756' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/4281613920"}]: dispatch 2024-04-22T21:41:06.006 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:41:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:41:06.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:05 smithi188 ceph-mon[22698]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:06.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:05 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1480794756' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/4281613920"}]': finished 2024-04-22T21:41:06.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:05 smithi188 ceph-mon[22698]: osdmap e94: 8 total, 8 up, 8 in 2024-04-22T21:41:06.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:05 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1424112759' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3912587609"}]: dispatch 2024-04-22T21:41:06.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:05 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3912587609"}]: dispatch 2024-04-22T21:41:06.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[21833]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:06.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1480794756' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/4281613920"}]': finished 2024-04-22T21:41:06.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[21833]: osdmap e94: 8 total, 8 up, 8 in 2024-04-22T21:41:06.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1424112759' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3912587609"}]: dispatch 2024-04-22T21:41:06.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3912587609"}]: dispatch 2024-04-22T21:41:06.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[26549]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:06.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1480794756' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:6801/4281613920"}]': finished 2024-04-22T21:41:06.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[26549]: osdmap e94: 8 total, 8 up, 8 in 2024-04-22T21:41:06.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1424112759' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3912587609"}]: dispatch 2024-04-22T21:41:06.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:05 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3912587609"}]: dispatch 2024-04-22T21:41:07.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:06 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3912587609"}]': finished 2024-04-22T21:41:07.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:06 smithi188 ceph-mon[22698]: osdmap e95: 8 total, 8 up, 8 in 2024-04-22T21:41:07.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:06 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/2126659669' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3416385233"}]: dispatch 2024-04-22T21:41:07.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:06 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3416385233"}]: dispatch 2024-04-22T21:41:07.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:06 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3912587609"}]': finished 2024-04-22T21:41:07.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:06 smithi165 ceph-mon[21833]: osdmap e95: 8 total, 8 up, 8 in 2024-04-22T21:41:07.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:06 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2126659669' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3416385233"}]: dispatch 2024-04-22T21:41:07.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:06 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3416385233"}]: dispatch 2024-04-22T21:41:07.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:06 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3912587609"}]': finished 2024-04-22T21:41:07.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:06 smithi165 ceph-mon[26549]: osdmap e95: 8 total, 8 up, 8 in 2024-04-22T21:41:07.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:06 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/2126659669' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3416385233"}]: dispatch 2024-04-22T21:41:07.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:06 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3416385233"}]: dispatch 2024-04-22T21:41:08.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:07 smithi188 ceph-mon[22698]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:08.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:07 smithi188 ceph-mon[22698]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3416385233"}]': finished 2024-04-22T21:41:08.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:07 smithi188 ceph-mon[22698]: osdmap e96: 8 total, 8 up, 8 in 2024-04-22T21:41:08.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:07 smithi165 ceph-mon[21833]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:08.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:07 smithi165 ceph-mon[21833]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3416385233"}]': finished 2024-04-22T21:41:08.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:07 smithi165 ceph-mon[21833]: osdmap e96: 8 total, 8 up, 8 in 2024-04-22T21:41:08.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:07 smithi165 ceph-mon[26549]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:08.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:07 smithi165 ceph-mon[26549]: from='client.? ' entity='client.iscsi.foo.smithi165.ceupac' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.165:0/3416385233"}]': finished 2024-04-22T21:41:08.058 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:07 smithi165 ceph-mon[26549]: osdmap e96: 8 total, 8 up, 8 in 2024-04-22T21:41:10.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:09 smithi188 ceph-mon[22698]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-22T21:41:10.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:09 smithi165 ceph-mon[21833]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-22T21:41:10.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:09 smithi165 ceph-mon[26549]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-22T21:41:12.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:11 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:12.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:11 smithi165 ceph-mon[26549]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:12.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:11 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:12.058 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:11 smithi165 ceph-mon[21833]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:12.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:11 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:12.257 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:11 smithi188 ceph-mon[22698]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:13.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:13 smithi188 ceph-mon[22698]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2024-04-22T21:41:13.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:13 smithi165 ceph-mon[21833]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2024-04-22T21:41:13.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:13 smithi165 ceph-mon[26549]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2024-04-22T21:41:14.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:41:13 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:13] "GET /metrics HTTP/1.1" 200 214859 "" "Prometheus/2.33.4" 2024-04-22T21:41:15.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:15 smithi188 ceph-mon[22698]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:15.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:15 smithi165 ceph-mon[21833]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:15.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:15 smithi165 ceph-mon[26549]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:16.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:41:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:41:17.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:17 smithi188 ceph-mon[22698]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-22T21:41:17.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:17 smithi165 ceph-mon[21833]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-22T21:41:17.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:17 smithi165 ceph-mon[26549]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-22T21:41:19.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:19 smithi188 ceph-mon[22698]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-04-22T21:41:19.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:19 smithi165 ceph-mon[21833]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-04-22T21:41:19.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:19 smithi165 ceph-mon[26549]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-04-22T21:41:21.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:21 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:21.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:21 smithi188 ceph-mon[22698]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:21.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:21 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:21.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:21 smithi165 ceph-mon[21833]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:21.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:21 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:21.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:21 smithi165 ceph-mon[26549]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:23.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:23 smithi188 ceph-mon[22698]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:23.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:23 smithi165 ceph-mon[21833]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:23.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:23 smithi165 ceph-mon[26549]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:24.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:41:23 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:23] "GET /metrics HTTP/1.1" 200 214812 "" "Prometheus/2.33.4" 2024-04-22T21:41:25.755 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:25 smithi165 ceph-mon[21833]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:25.755 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:25 smithi165 ceph-mon[26549]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:25.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:25 smithi188 ceph-mon[22698]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:26.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:41:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:41:27.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:27 smithi188 ceph-mon[22698]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:27.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:27 smithi165 ceph-mon[26549]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:27.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:27 smithi165 ceph-mon[21833]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:29.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:29 smithi188 ceph-mon[22698]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:29.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:29 smithi165 ceph-mon[21833]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:29.897 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:29 smithi165 ceph-mon[26549]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:31.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:31 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:31.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:31 smithi188 ceph-mon[22698]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:31.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:31 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:31.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:31 smithi165 ceph-mon[21833]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:31.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:31 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:31.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:31 smithi165 ceph-mon[26549]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:33.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:33 smithi188 ceph-mon[22698]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:33.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:33 smithi165 ceph-mon[26549]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:33.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:33 smithi165 ceph-mon[21833]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:34.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:41:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:33] "GET /metrics HTTP/1.1" 200 214812 "" "Prometheus/2.33.4" 2024-04-22T21:41:35.557 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:35 smithi165 ceph-mon[21833]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:35.557 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:35 smithi165 ceph-mon[26549]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:35.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:35 smithi188 ceph-mon[22698]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:36.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:41:35 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:41:37.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:37 smithi188 ceph-mon[22698]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:37.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:37 smithi165 ceph-mon[21833]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:37.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:37 smithi165 ceph-mon[26549]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:38.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:41:38.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:41:38.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:41:38.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:41:38.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:41:38.808 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:41:39.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:39 smithi188 ceph-mon[22698]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:39.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:39 smithi165 ceph-mon[21833]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:39.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:39 smithi165 ceph-mon[26549]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:41.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:41 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:41.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:41 smithi188 ceph-mon[22698]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:41.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:41 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:41.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:41 smithi165 ceph-mon[21833]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:41.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:41 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:41.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:41 smithi165 ceph-mon[26549]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:43.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:43 smithi188 ceph-mon[22698]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:43.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:43 smithi165 ceph-mon[21833]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:43.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:43 smithi165 ceph-mon[26549]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:44.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:41:43 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:43] "GET /metrics HTTP/1.1" 200 214774 "" "Prometheus/2.33.4" 2024-04-22T21:41:45.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:45 smithi188 ceph-mon[22698]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:45.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:45 smithi165 ceph-mon[21833]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:45.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:45 smithi165 ceph-mon[26549]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:46.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:41:45 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:41:47.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:47 smithi188 ceph-mon[22698]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:47.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:47 smithi165 ceph-mon[21833]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:47.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:47 smithi165 ceph-mon[26549]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:49.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:49 smithi188 ceph-mon[22698]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:49.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:49 smithi165 ceph-mon[21833]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:49.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:49 smithi165 ceph-mon[26549]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:51.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:51 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:51.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:51 smithi188 ceph-mon[22698]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:51.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:51 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:51.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:51 smithi165 ceph-mon[21833]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:51.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:51 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:41:51.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:51 smithi165 ceph-mon[26549]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:53.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:53 smithi188 ceph-mon[22698]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:53.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:53 smithi165 ceph-mon[21833]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:53.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:53 smithi165 ceph-mon[26549]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:54.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:41:53 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:53] "GET /metrics HTTP/1.1" 200 214779 "" "Prometheus/2.33.4" 2024-04-22T21:41:55.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:55 smithi188 ceph-mon[22698]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:55.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:55 smithi165 ceph-mon[21833]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:55.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:55 smithi165 ceph-mon[26549]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:41:56.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:41:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:41:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:41:57.746 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:57 smithi188 ceph-mon[22698]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:57.746 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:57 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:41:57.746 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:57 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:41:57.752 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:57 smithi165 ceph-mon[21833]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:57.752 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:57 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:41:57.752 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:57 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:41:57.753 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:57 smithi165 ceph-mon[26549]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:41:57.753 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:57 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:41:57.753 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:57 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:41:59.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:58 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:41:59.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:58 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:41:59.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:58 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:00.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:41:59 smithi165 ceph-mon[21833]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:00.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:41:59 smithi165 ceph-mon[26549]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:00.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:41:59 smithi188 ceph-mon[22698]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:02.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:01 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:02.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:01 smithi165 ceph-mon[26549]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:02.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:01 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:02.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:01 smithi165 ceph-mon[21833]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:02.059 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:01 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:02.059 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:01 smithi188 ceph-mon[22698]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:03.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:03 smithi188 ceph-mon[22698]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:03.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:03 smithi165 ceph-mon[21833]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:03.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:03 smithi165 ceph-mon[26549]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:04.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:42:03 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:03] "GET /metrics HTTP/1.1" 200 214779 "" "Prometheus/2.33.4" 2024-04-22T21:42:05.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:05 smithi188 ceph-mon[22698]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:05.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:05 smithi165 ceph-mon[21833]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:05.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:05 smithi165 ceph-mon[26549]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:06.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:42:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:42:07.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:07 smithi188 ceph-mon[22698]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:07.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:07 smithi165 ceph-mon[21833]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:07.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:07 smithi165 ceph-mon[26549]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:09.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:09 smithi188 ceph-mon[22698]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:09.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:09 smithi165 ceph-mon[21833]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:09.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:09 smithi165 ceph-mon[26549]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:11.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:11 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:11.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:11 smithi188 ceph-mon[22698]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:11.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:11 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:11.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:11 smithi165 ceph-mon[26549]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:11.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:11 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:11.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:11 smithi165 ceph-mon[21833]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:13.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:13 smithi188 ceph-mon[22698]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:13.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:13 smithi165 ceph-mon[21833]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:13.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:13 smithi165 ceph-mon[26549]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:14.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:42:13 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:13] "GET /metrics HTTP/1.1" 200 214770 "" "Prometheus/2.33.4" 2024-04-22T21:42:15.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:15 smithi188 ceph-mon[22698]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:15.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:15 smithi165 ceph-mon[21833]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:15.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:15 smithi165 ceph-mon[26549]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:16.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:42:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:42:17.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:17 smithi188 ceph-mon[22698]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:17.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:17 smithi165 ceph-mon[21833]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:17.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:17 smithi165 ceph-mon[26549]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:19.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:19 smithi188 ceph-mon[22698]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:19.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:19 smithi165 ceph-mon[21833]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:19.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:19 smithi165 ceph-mon[26549]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:21.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:21 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:21.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:21 smithi188 ceph-mon[22698]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:21.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:21 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:21.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:21 smithi165 ceph-mon[21833]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:21.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:21 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:21.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:21 smithi165 ceph-mon[26549]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:23.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:23 smithi188 ceph-mon[22698]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:23.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:23 smithi165 ceph-mon[21833]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:23.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:23 smithi165 ceph-mon[26549]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:24.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:42:23 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:23] "GET /metrics HTTP/1.1" 200 214774 "" "Prometheus/2.33.4" 2024-04-22T21:42:25.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:25 smithi188 ceph-mon[22698]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:25.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:25 smithi165 ceph-mon[21833]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:25.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:25 smithi165 ceph-mon[26549]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:26.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:42:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:42:27.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:27 smithi188 ceph-mon[22698]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:27.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:27 smithi165 ceph-mon[21833]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:27.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:27 smithi165 ceph-mon[26549]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:29.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:29 smithi188 ceph-mon[22698]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:29.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:29 smithi165 ceph-mon[21833]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:29.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:29 smithi165 ceph-mon[26549]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:31.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:31 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:31.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:31 smithi188 ceph-mon[22698]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:31.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:31 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:31.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:31 smithi165 ceph-mon[21833]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:31.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:31 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:31.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:31 smithi165 ceph-mon[26549]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:33.759 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:33 smithi188 ceph-mon[22698]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:33.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:33 smithi165 ceph-mon[21833]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:33.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:33 smithi165 ceph-mon[26549]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:34.271 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:42:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:33] "GET /metrics HTTP/1.1" 200 214774 "" "Prometheus/2.33.4" 2024-04-22T21:42:35.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:35 smithi188 ceph-mon[22698]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:35.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:35 smithi165 ceph-mon[26549]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:35.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:35 smithi165 ceph-mon[21833]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:36.258 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:42:35 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:42:37.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:37 smithi188 ceph-mon[22698]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:37.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:37 smithi165 ceph-mon[26549]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:37.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:37 smithi165 ceph-mon[21833]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:38.753 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:42:38.753 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:38 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:42:38.753 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:42:38.753 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:38 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:42:38.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:42:38.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:38 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:42:38.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T21:42:39.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:39 smithi188 ceph-mon[22698]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:39.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:39 smithi165 ceph-mon[21833]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:39.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:39 smithi165 ceph-mon[26549]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:41.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T21:42:41.737 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:41 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:41.738 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:41 smithi165 ceph-mon[21833]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:41.738 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:41 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:41.738 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:41 smithi165 ceph-mon[26549]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:41.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:41 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:41.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:41 smithi188 ceph-mon[22698]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:43.662 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:43 smithi165 ceph-mon[21833]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:43.662 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:43 smithi165 ceph-mon[26549]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:43.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:43 smithi188 ceph-mon[22698]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:43.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T21:42:44.288 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:42:43 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:43] "GET /metrics HTTP/1.1" 200 214783 "" "Prometheus/2.33.4" 2024-04-22T21:42:45.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:45 smithi188 ceph-mon[22698]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:45.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:45 smithi165 ceph-mon[21833]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:45.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:45 smithi165 ceph-mon[26549]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:42:46.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:42:45 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:42:46.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-22T21:42:47.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:47 smithi188 ceph-mon[22698]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:47.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:47 smithi165 ceph-mon[21833]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:47.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:47 smithi165 ceph-mon[26549]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:48.569 INFO:teuthology.orchestra.run.smithi165.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:42:49.180 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T21:42:49.194 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi165.front.sepia.ceph.com 2024-04-22T21:42:49.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-22T21:42:49.758 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:42:49.455+0000 7f5b751fd700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T21:42:49.759 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:49.759 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:49.759 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:42:49.759 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:49.759 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:42:49.759 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:42:49.759 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:49.759 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:42:49.759 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:49.760 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:42:49.761 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:42:49.761 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:50.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:42:50.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:42:50.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:50.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:50 smithi165 ceph-mon[26549]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T21:42:50.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:50 smithi165 ceph-mon[26549]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-22T21:42:50.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:50 smithi165 ceph-mon[26549]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T21:42:50.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:50 smithi165 ceph-mon[21833]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T21:42:50.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:50 smithi165 ceph-mon[21833]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-22T21:42:50.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:50 smithi165 ceph-mon[21833]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:50 smithi188 ceph-mon[22698]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:50 smithi188 ceph-mon[22698]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-22T21:42:51.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:50 smithi188 ceph-mon[22698]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T21:42:51.210 INFO:teuthology.orchestra.run.smithi165.stdout:true 2024-04-22T21:42:51.508 INFO:teuthology.orchestra.run.smithi165.stdout:"Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-22T21:42:51.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:51 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:51.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:51 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:52.007 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:51 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:42:52.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch ps' 2024-04-22T21:42:52.772 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:52 smithi165 ceph-mon[21833]: from='client.24928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:52.772 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:52 smithi165 ceph-mon[21833]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:52.773 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:52 smithi165 ceph-mon[21833]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:52.773 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:52 smithi165 ceph-mon[26549]: from='client.24928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:52.773 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:52 smithi165 ceph-mon[26549]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:52.773 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:52 smithi165 ceph-mon[26549]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:53.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:52 smithi188 ceph-mon[22698]: from='client.24928 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:53.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:52 smithi188 ceph-mon[22698]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:53.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:52 smithi188 ceph-mon[22698]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:53.952 INFO:teuthology.orchestra.run.smithi165.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:alertmanager.a smithi165 *:9093,9094 running (2m) 116s ago 2m 12.9M - ba2b418f427c 5f88b3382f58 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:grafana.a smithi188 *:3000 running (2m) 116s ago 2m 48.3M - 8.3.5 dad864ee21e9 1184bc13bc40 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:iscsi.foo.smithi165.ceupac smithi165 running (2m) 116s ago 2m 70.6M - 3.5 e1d6a67b021e 50a7513b09e7 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:mgr.x smithi188 *:8443 running (7m) 116s ago 7m 410M - 17.2.0 e1d6a67b021e 9f10aded29b4 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:mgr.y smithi165 *:9283 running (9m) 116s ago 9m 456M - 17.2.0 e1d6a67b021e edb11b64e074 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:mon.a smithi165 running (9m) 116s ago 9m 50.2M 2048M 17.2.0 e1d6a67b021e b2377729385b 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:mon.b smithi188 running (7m) 116s ago 7m 41.3M 2048M 17.2.0 e1d6a67b021e 1aa8711a0f5e 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:mon.c smithi165 running (8m) 116s ago 8m 38.5M 2048M 17.2.0 e1d6a67b021e 8798a7ddca56 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:node-exporter.a smithi165 *:9100 running (3m) 116s ago 3m 15.7M - 1dbe0e931976 f4355268ef2f 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:node-exporter.b smithi188 *:9100 running (3m) 116s ago 3m 13.7M - 1dbe0e931976 211a953ad0ee 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0 smithi165 running (7m) 116s ago 7m 49.2M 4016M 17.2.0 e1d6a67b021e b414e5f11e45 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1 smithi165 running (7m) 116s ago 6m 49.9M 4016M 17.2.0 e1d6a67b021e 12a0b8284d72 2024-04-22T21:42:53.953 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2 smithi165 running (6m) 116s ago 6m 45.6M 4016M 17.2.0 e1d6a67b021e 84c9568b476f 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3 smithi165 running (6m) 116s ago 6m 46.7M 4016M 17.2.0 e1d6a67b021e 8ea698e06b17 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4 smithi188 running (5m) 116s ago 5m 47.3M 4272M 17.2.0 e1d6a67b021e a9f173f084db 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5 smithi188 running (5m) 116s ago 5m 46.4M 4272M 17.2.0 e1d6a67b021e a56baa2e0a10 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6 smithi188 running (5m) 116s ago 5m 44.5M 4272M 17.2.0 e1d6a67b021e 1b040381578e 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7 smithi188 running (4m) 116s ago 4m 48.0M 4272M 17.2.0 e1d6a67b021e 41502d6e8698 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:prometheus.a smithi188 *:9095 running (2m) 116s ago 3m 44.6M - 514e6a882f6e b1b3a948cedf 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:rgw.foo.smithi165.pbstvw smithi165 *:8000 running (2m) 116s ago 2m 70.4M - 17.2.0 e1d6a67b021e a5ba817f72be 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:rgw.foo.smithi188.wgqixq smithi188 *:8000 running (2m) 116s ago 2m 68.9M - 17.2.0 e1d6a67b021e fd729a990209 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:rgw.smpl.smithi165.hjvsmr smithi165 *:80 running (2m) 116s ago 2m 71.5M - 17.2.0 e1d6a67b021e 9cb58837a922 2024-04-22T21:42:53.954 INFO:teuthology.orchestra.run.smithi165.stdout:rgw.smpl.smithi188.ryxylz smithi188 *:80 running (2m) 116s ago 2m 69.2M - 17.2.0 e1d6a67b021e e49cb18c9ec3 2024-04-22T21:42:54.274 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:42:53 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:53] "GET /metrics HTTP/1.1" 200 214847 "" "Prometheus/2.33.4" 2024-04-22T21:42:54.527 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:54 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:54.527 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:54 smithi165 ceph-mon[21833]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:54.527 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:54 smithi165 ceph-mon[21833]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:54.527 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:54 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:54.528 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:54 smithi165 ceph-mon[26549]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:54.528 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:54 smithi165 ceph-mon[26549]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:54.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions' 2024-04-22T21:42:54.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:54 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:42:54.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:54 smithi188 ceph-mon[22698]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:54.758 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:54 smithi188 ceph-mon[22698]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:42:56.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:42:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:42:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:42:56.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:56 smithi188 ceph-mon[22698]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:56.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:56 smithi165 ceph-mon[21833]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:56.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:56 smithi165 ceph-mon[26549]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:56.822 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-04-22T21:42:56.822 INFO:teuthology.orchestra.run.smithi165.stdout: "mon": { 2024-04-22T21:42:56.822 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-22T21:42:56.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr": { 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": { 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: "mds": {}, 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw": { 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: "overall": { 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:42:56.823 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-04-22T21:42:57.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-22T21:42:57.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:57 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/1078798919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:42:57.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:57 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/1078798919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:42:57.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:57 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/1078798919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:42:58.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:42:58 smithi188 ceph-mon[22698]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:58.778 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:42:58 smithi165 ceph-mon[21833]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:58.779 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:42:58 smithi165 ceph-mon[26549]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-04-22T21:42:59.248 INFO:teuthology.orchestra.run.smithi165.stdout:wait for servicemap items w/ changing names to refresh 2024-04-22T21:42:59.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'sleep 60' 2024-04-22T21:43:00.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:00 smithi188 ceph-mon[22698]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-22T21:43:00.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:00 smithi165 ceph-mon[21833]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-22T21:43:00.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:00 smithi165 ceph-mon[26549]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-22T21:43:01.519 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:01 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:01.520 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:01 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:02.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:01 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:02.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:02 smithi165 ceph-mon[26549]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:02.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:02 smithi165 ceph-mon[21833]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:03.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:02 smithi188 ceph-mon[22698]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:04.306 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:43:03 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:03] "GET /metrics HTTP/1.1" 200 214847 "" "Prometheus/2.33.4" 2024-04-22T21:43:04.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:04 smithi165 ceph-mon[26549]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:04.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:04 smithi165 ceph-mon[21833]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:04.808 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:04 smithi188 ceph-mon[22698]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:06.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:43:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:43:06.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:06 smithi165 ceph-mon[21833]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:06.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:06 smithi165 ceph-mon[26549]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:07.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:06 smithi188 ceph-mon[22698]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:08.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:08 smithi188 ceph-mon[22698]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:08.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:08 smithi165 ceph-mon[21833]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:08.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:08 smithi165 ceph-mon[26549]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:10.556 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:10 smithi165 ceph-mon[21833]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:10.557 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:10 smithi165 ceph-mon[26549]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:10.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:10 smithi188 ceph-mon[22698]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:11.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:11 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:11.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:11 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:12.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:11 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:12.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:12 smithi165 ceph-mon[26549]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:12.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:12 smithi165 ceph-mon[21833]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:13.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:12 smithi188 ceph-mon[22698]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:14.306 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:43:13 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:13] "GET /metrics HTTP/1.1" 200 214826 "" "Prometheus/2.33.4" 2024-04-22T21:43:14.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:14 smithi165 ceph-mon[26549]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:14.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:14 smithi165 ceph-mon[21833]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:15.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:14 smithi188 ceph-mon[22698]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:16.258 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:43:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:43:16.810 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:16 smithi165 ceph-mon[21833]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:16.811 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:16 smithi165 ceph-mon[26549]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:17.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:16 smithi188 ceph-mon[22698]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:18.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:18 smithi188 ceph-mon[22698]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:18.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:18 smithi165 ceph-mon[21833]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:18.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:18 smithi165 ceph-mon[26549]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:20.556 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:20 smithi165 ceph-mon[21833]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:20.557 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:20 smithi165 ceph-mon[26549]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:21.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:20 smithi188 ceph-mon[22698]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:21.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:21 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:21.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:21 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:22.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:21 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:22.810 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:22 smithi165 ceph-mon[21833]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:22.810 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:22 smithi165 ceph-mon[26549]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:23.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:22 smithi188 ceph-mon[22698]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:24.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:43:23 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:23] "GET /metrics HTTP/1.1" 200 214792 "" "Prometheus/2.33.4" 2024-04-22T21:43:24.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:24 smithi165 ceph-mon[21833]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:24.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:24 smithi165 ceph-mon[26549]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:25.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:24 smithi188 ceph-mon[22698]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:26.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:43:25 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:25] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:43:26.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:26 smithi165 ceph-mon[21833]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:26.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:26 smithi165 ceph-mon[26549]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:27.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:26 smithi188 ceph-mon[22698]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:28.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:28 smithi188 ceph-mon[22698]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-22T21:43:28.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:28 smithi165 ceph-mon[21833]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-22T21:43:28.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:28 smithi165 ceph-mon[26549]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-22T21:43:30.556 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:30 smithi165 ceph-mon[21833]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:30.557 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:30 smithi165 ceph-mon[26549]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:31.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:30 smithi188 ceph-mon[22698]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:31.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:31 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:31.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:31 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:32.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:31 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:32.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:32 smithi165 ceph-mon[21833]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-22T21:43:32.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:32 smithi165 ceph-mon[26549]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-22T21:43:33.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:32 smithi188 ceph-mon[22698]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-22T21:43:34.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:43:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:33] "GET /metrics HTTP/1.1" 200 214792 "" "Prometheus/2.33.4" 2024-04-22T21:43:34.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:34 smithi165 ceph-mon[21833]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-22T21:43:34.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:34 smithi165 ceph-mon[26549]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-22T21:43:34.808 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:34 smithi188 ceph-mon[22698]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-22T21:43:36.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:43:35 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:43:36.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:36 smithi165 ceph-mon[21833]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:36.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:36 smithi165 ceph-mon[26549]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:37.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:36 smithi188 ceph-mon[22698]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:38.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:38 smithi188 ceph-mon[22698]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:38.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:38 smithi165 ceph-mon[21833]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:38.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:38 smithi165 ceph-mon[26549]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:39.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:43:39.757 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:39 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:43:39.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:43:39.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:39 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:43:39.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T21:43:39.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:39 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T21:43:40.556 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:40 smithi165 ceph-mon[21833]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:40.557 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:40 smithi165 ceph-mon[26549]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:40.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:40 smithi188 ceph-mon[22698]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:41.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:41 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:41.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:41 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:41.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:41 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:42.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:42 smithi188 ceph-mon[22698]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:42.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:42 smithi165 ceph-mon[21833]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:42.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:42 smithi165 ceph-mon[26549]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:44.306 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:43:43 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:43] "GET /metrics HTTP/1.1" 200 214791 "" "Prometheus/2.33.4" 2024-04-22T21:43:44.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:44 smithi165 ceph-mon[21833]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:44.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:44 smithi165 ceph-mon[26549]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:45.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:44 smithi188 ceph-mon[22698]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:46.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:43:45 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:45] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:43:46.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:46 smithi165 ceph-mon[21833]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:46.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:46 smithi165 ceph-mon[26549]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:47.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:46 smithi188 ceph-mon[22698]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:48.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:48 smithi188 ceph-mon[22698]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:48.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:48 smithi165 ceph-mon[21833]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:48.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:48 smithi165 ceph-mon[26549]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:49.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:43:49.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:49 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:43:49.768 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:43:49.768 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:49 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:43:49.768 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T21:43:49.768 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:49 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T21:43:51.056 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:50 smithi165 ceph-mon[26549]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:51.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:50 smithi165 ceph-mon[26549]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:43:51.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:50 smithi165 ceph-mon[21833]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:51.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:50 smithi165 ceph-mon[21833]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:43:51.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:50 smithi188 ceph-mon[22698]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:51.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:50 smithi188 ceph-mon[22698]: from='mgr.24355 172.21.15.165:0/3030822572' entity='mgr.y' 2024-04-22T21:43:52.056 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:51 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:52.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:51 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:52.259 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:51 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:43:53.056 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:52 smithi165 ceph-mon[21833]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:53.056 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:52 smithi165 ceph-mon[26549]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:53.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:52 smithi188 ceph-mon[22698]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:54.306 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:43:53 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:53] "GET /metrics HTTP/1.1" 200 214782 "" "Prometheus/2.33.4" 2024-04-22T21:43:55.056 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:54 smithi165 ceph-mon[21833]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:55.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:54 smithi165 ceph-mon[26549]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:55.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:54 smithi188 ceph-mon[22698]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:56.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:43:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:43:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:43:57.057 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:56 smithi165 ceph-mon[21833]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:57.057 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:56 smithi165 ceph-mon[26549]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:57.256 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:56 smithi188 ceph-mon[22698]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:43:58.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:43:58 smithi188 ceph-mon[22698]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:58.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:43:58 smithi165 ceph-mon[21833]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:43:58.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:43:58 smithi165 ceph-mon[26549]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:00.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:00 smithi165 ceph-mon[21833]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:00.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:00 smithi165 ceph-mon[26549]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:01.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:00 smithi188 ceph-mon[22698]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:01.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:01 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:44:01.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:01 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:44:02.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:01 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:44:02.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch ps' 2024-04-22T21:44:02.673 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:02 smithi165 ceph-mon[21833]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:02.673 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:02 smithi165 ceph-mon[26549]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:03.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:02 smithi188 ceph-mon[22698]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:04.027 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:44:03 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:44:03] "GET /metrics HTTP/1.1" 200 214782 "" "Prometheus/2.33.4" 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:alertmanager.a smithi165 *:9093,9094 running (3m) 3m ago 4m 12.9M - ba2b418f427c 5f88b3382f58 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:grafana.a smithi188 *:3000 running (3m) 3m ago 3m 48.3M - 8.3.5 dad864ee21e9 1184bc13bc40 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:iscsi.foo.smithi165.ceupac smithi165 running (3m) 3m ago 3m 70.6M - 3.5 e1d6a67b021e 50a7513b09e7 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:mgr.x smithi188 *:8443 running (8m) 3m ago 8m 410M - 17.2.0 e1d6a67b021e 9f10aded29b4 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:mgr.y smithi165 *:9283 running (10m) 3m ago 10m 456M - 17.2.0 e1d6a67b021e edb11b64e074 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:mon.a smithi165 running (10m) 3m ago 10m 50.2M 2048M 17.2.0 e1d6a67b021e b2377729385b 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:mon.b smithi188 running (9m) 3m ago 9m 41.3M 2048M 17.2.0 e1d6a67b021e 1aa8711a0f5e 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:mon.c smithi165 running (9m) 3m ago 9m 38.5M 2048M 17.2.0 e1d6a67b021e 8798a7ddca56 2024-04-22T21:44:04.429 INFO:teuthology.orchestra.run.smithi165.stdout:node-exporter.a smithi165 *:9100 running (5m) 3m ago 5m 15.7M - 1dbe0e931976 f4355268ef2f 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:node-exporter.b smithi188 *:9100 running (4m) 3m ago 4m 13.7M - 1dbe0e931976 211a953ad0ee 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0 smithi165 running (8m) 3m ago 8m 49.2M 4016M 17.2.0 e1d6a67b021e b414e5f11e45 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1 smithi165 running (8m) 3m ago 8m 49.9M 4016M 17.2.0 e1d6a67b021e 12a0b8284d72 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2 smithi165 running (7m) 3m ago 7m 45.6M 4016M 17.2.0 e1d6a67b021e 84c9568b476f 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3 smithi165 running (7m) 3m ago 7m 46.7M 4016M 17.2.0 e1d6a67b021e 8ea698e06b17 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:osd.4 smithi188 running (6m) 3m ago 6m 47.3M 4272M 17.2.0 e1d6a67b021e a9f173f084db 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:osd.5 smithi188 running (6m) 3m ago 6m 46.4M 4272M 17.2.0 e1d6a67b021e a56baa2e0a10 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:osd.6 smithi188 running (6m) 3m ago 6m 44.5M 4272M 17.2.0 e1d6a67b021e 1b040381578e 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:osd.7 smithi188 running (5m) 3m ago 5m 48.0M 4272M 17.2.0 e1d6a67b021e 41502d6e8698 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:prometheus.a smithi188 *:9095 running (3m) 3m ago 5m 44.6M - 514e6a882f6e b1b3a948cedf 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:rgw.foo.smithi165.pbstvw smithi165 *:8000 running (3m) 3m ago 3m 70.4M - 17.2.0 e1d6a67b021e a5ba817f72be 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:rgw.foo.smithi188.wgqixq smithi188 *:8000 running (3m) 3m ago 3m 68.9M - 17.2.0 e1d6a67b021e fd729a990209 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:rgw.smpl.smithi165.hjvsmr smithi165 *:80 running (3m) 3m ago 3m 71.5M - 17.2.0 e1d6a67b021e 9cb58837a922 2024-04-22T21:44:04.430 INFO:teuthology.orchestra.run.smithi165.stdout:rgw.smpl.smithi188.ryxylz smithi188 *:80 running (3m) 3m ago 3m 69.2M - 17.2.0 e1d6a67b021e e49cb18c9ec3 2024-04-22T21:44:04.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:04 smithi165 ceph-mon[26549]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:04.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:04 smithi165 ceph-mon[21833]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:04.808 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:04 smithi188 ceph-mon[22698]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:05.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions' 2024-04-22T21:44:05.622 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:05 smithi165 ceph-mon[21833]: from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:44:05.622 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:05 smithi165 ceph-mon[26549]: from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:44:05.828 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:05 smithi188 ceph-mon[22698]: from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:44:06.217 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:44:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:44:05] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:44:06.759 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:06 smithi165 ceph-mon[21833]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:06.760 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:06 smithi165 ceph-mon[26549]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:06.949 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:06 smithi188 ceph-mon[22698]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:07.229 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-04-22T21:44:07.230 INFO:teuthology.orchestra.run.smithi165.stdout: "mon": { 2024-04-22T21:44:07.230 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-22T21:44:07.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:44:07.230 INFO:teuthology.orchestra.run.smithi165.stdout: "mgr": { 2024-04-22T21:44:07.230 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T21:44:07.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:44:07.231 INFO:teuthology.orchestra.run.smithi165.stdout: "osd": { 2024-04-22T21:44:07.231 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-22T21:44:07.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:44:07.231 INFO:teuthology.orchestra.run.smithi165.stdout: "mds": {}, 2024-04-22T21:44:07.231 INFO:teuthology.orchestra.run.smithi165.stdout: "rgw": { 2024-04-22T21:44:07.232 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-22T21:44:07.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-04-22T21:44:07.232 INFO:teuthology.orchestra.run.smithi165.stdout: "overall": { 2024-04-22T21:44:07.232 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-22T21:44:07.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-04-22T21:44:07.232 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-04-22T21:44:07.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:07 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/4151481051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:44:07.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:07 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/4151481051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:44:07.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch upgrade status' 2024-04-22T21:44:08.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:07 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/4151481051' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:44:08.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:08 smithi165 ceph-mon[21833]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:08.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:08 smithi165 ceph-mon[26549]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:09.005 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:08 smithi188 ceph-mon[22698]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:10.087 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-04-22T21:44:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", 2024-04-22T21:44:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "in_progress": true, 2024-04-22T21:44:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "services_complete": [], 2024-04-22T21:44:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "progress": "", 2024-04-22T21:44:10.088 INFO:teuthology.orchestra.run.smithi165.stdout: "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-22T21:44:10.088 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-04-22T21:44:10.660 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:10 smithi165 ceph-mon[21833]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:10.660 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:10 smithi165 ceph-mon[21833]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:44:10.660 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:10 smithi165 ceph-mon[26549]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:10.660 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:10 smithi165 ceph-mon[26549]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:44:10.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph health detail' 2024-04-22T21:44:11.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:10 smithi188 ceph-mon[22698]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:11.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:10 smithi188 ceph-mon[22698]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T21:44:11.699 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:11 smithi165 ceph-mon[21833]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:44:11.700 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:11 smithi165 ceph-mon[26549]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:44:12.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:11 smithi188 ceph-mon[22698]: from='client.24763 -' entity='client.iscsi.foo.smithi165.ceupac' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T21:44:12.746 INFO:teuthology.orchestra.run.smithi165.stdout:HEALTH_WARN Upgrade: failed to pull target image 2024-04-22T21:44:12.747 INFO:teuthology.orchestra.run.smithi165.stdout:[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T21:44:12.747 INFO:teuthology.orchestra.run.smithi165.stdout: host smithi165 `cephadm pull` failed: cephadm exited with an error code: 1, stderr: Pulling container image quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f... 2024-04-22T21:44:12.747 INFO:teuthology.orchestra.run.smithi165.stdout:Non-zero exit code 125 from /bin/podman pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:44:12.747 INFO:teuthology.orchestra.run.smithi165.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f... 2024-04-22T21:44:12.747 INFO:teuthology.orchestra.run.smithi165.stdout:/bin/podman: stderr Error: initializing source docker://quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f: reading manifest 430e09df97c8fc7dc2b2ae424f68ed11366c540f in quay.ceph.io/ceph-ci/ceph: manifest unknown 2024-04-22T21:44:12.747 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: Failed command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:44:12.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:12 smithi165 ceph-mon[21833]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:12.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:12 smithi165 ceph-mon[26549]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:13.005 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:12 smithi188 ceph-mon[22698]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:13.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-22T21:44:13.801 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:13 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2915360346' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T21:44:13.802 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:13 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/2915360346' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T21:44:14.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:13 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/2915360346' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T21:44:14.056 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:44:13 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:44:13] "GET /metrics HTTP/1.1" 200 214793 "" "Prometheus/2.33.4" 2024-04-22T21:44:14.845 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:14 smithi165 ceph-mon[21833]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:14.845 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:14 smithi165 ceph-mon[26549]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:15.006 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:14 smithi188 ceph-mon[22698]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:15.515 INFO:teuthology.orchestra.run.smithi165.stdout:true 2024-04-22T21:44:15.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:15 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2131667862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:44:15.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:15 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/2131667862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:44:15.948 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:15 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/2131667862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:44:16.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-22T21:44:16.256 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:44:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x[23771]: ::ffff:172.21.15.188 - - [22/Apr/2024:21:44:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T21:44:16.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:16 smithi188 ceph-mon[22698]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:16.806 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:16 smithi165 ceph-mon[21833]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:16.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:16 smithi165 ceph-mon[26549]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T21:44:18.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:18 smithi188 ceph-mon[22698]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:18.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:18 smithi188 ceph-mon[22698]: from='client.? 172.21.15.165:0/2590214446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:44:18.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:18 smithi165 ceph-mon[21833]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:18.807 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:18 smithi165 ceph-mon[21833]: from='client.? 172.21.15.165:0/2590214446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:44:18.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:18 smithi165 ceph-mon[26549]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T21:44:18.807 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:18 smithi165 ceph-mon[26549]: from='client.? 172.21.15.165:0/2590214446' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T21:44:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:44:19.101 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi165 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-22T21:44:19.345 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4d0b013d53194c8b972ba2500031f18c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi165 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-22T21:44:19.347 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T21:44:19.357 INFO:tasks.cephadm:Teardown begin 2024-04-22T21:44:19.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:44:19.390 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:44:19.422 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T21:44:19.422 DEBUG:teuthology.orchestra.run.smithi165:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T21:44:19.446 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T21:44:19.479 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T21:44:19.480 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-22T21:44:19.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.a 2024-04-22T21:44:19.682 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:19 smithi165 systemd[1]: Stopping Ceph mon.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:19.682 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:19 smithi165 bash[54300]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-22T21:44:19.973 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:19 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a[21829]: 2024-04-22T21:44:19.701+0000 7ff3f06ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:44:19.973 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:19 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a[21829]: 2024-04-22T21:44:19.701+0000 7ff3f06ba700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T21:44:19.974 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:19 smithi165 podman[54313]: 2024-04-22 21:44:19.732516344 +0000 UTC m=+0.151634482 container died b2377729385b94128255e6ac38bcefaba4425e89cc321f74a1a3cb04c0684e51 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, io.openshift.expose-services=, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, release=754, ceph=True, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T21:44:20.229 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:20 smithi165 podman[54313]: 2024-04-22 21:44:20.210367639 +0000 UTC m=+0.629485782 container cleanup b2377729385b94128255e6ac38bcefaba4425e89cc321f74a1a3cb04c0684e51 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a, RELEASE=HEAD, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, name=centos-stream, io.openshift.expose-services=, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T21:44:20.230 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:20 smithi165 bash[54313]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-a 2024-04-22T21:44:20.230 INFO:journalctl@ceph.mon.a.smithi165.stdout:Apr 22 21:44:20 smithi165 bash[54398]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-22T21:44:20.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.a.service' 2024-04-22T21:44:20.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:44:20.526 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-22T21:44:20.526 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-22T21:44:20.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.c 2024-04-22T21:44:20.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:20 smithi165 systemd[1]: Stopping Ceph mon.c for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:21.054 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:20 smithi165 bash[54452]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon.c" found: no such container 2024-04-22T21:44:21.054 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:20 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-c[26545]: 2024-04-22T21:44:20.748+0000 7f481a3b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:44:21.054 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:20 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-c[26545]: 2024-04-22T21:44:20.748+0000 7f481a3b7700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-22T21:44:21.054 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:20 smithi165 podman[54462]: 2024-04-22 21:44:20.799658833 +0000 UTC m=+0.186908646 container died 8798a7ddca5671eda62940694f9b3d5ac98c3e4414a72983ab5d747d5cd12746 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-c, version=8, distribution-scope=public, name=centos-stream, io.buildah.version=1.19.8, GIT_CLEAN=True, GIT_BRANCH=HEAD, vcs-type=git, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T21:44:21.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:21 smithi165 podman[54462]: 2024-04-22 21:44:21.281806912 +0000 UTC m=+0.669056732 container cleanup 8798a7ddca5671eda62940694f9b3d5ac98c3e4414a72983ab5d747d5cd12746 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-c, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, distribution-scope=public, release=754, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_CLEAN=True, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T21:44:21.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:21 smithi165 bash[54462]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-c 2024-04-22T21:44:21.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Apr 22 21:44:21 smithi165 bash[54489]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon.c" found: no such container 2024-04-22T21:44:21.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.c.service' 2024-04-22T21:44:21.545 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:44:21.546 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-22T21:44:21.546 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-22T21:44:21.546 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.b 2024-04-22T21:44:21.905 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:21 smithi188 systemd[1]: Stopping Ceph mon.b for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:21.905 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:21 smithi188 bash[43458]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon.b" found: no such container 2024-04-22T21:44:22.159 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:21 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-b[22672]: 2024-04-22T21:44:21.924+0000 7fb14c38f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:44:22.160 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:21 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-b[22672]: 2024-04-22T21:44:21.924+0000 7fb14c38f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-22T21:44:22.438 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:22 smithi188 podman[43471]: 2024-04-22 21:44:22.179906266 +0000 UTC m=+0.528152386 container died 1aa8711a0f5e7dec13b36be36f3270e20bb63c8489927488977193a98074f41d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-b, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, ceph=True, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., vcs-type=git, io.openshift.expose-services=, com.redhat.component=centos-stream-container, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, release=754, GIT_BRANCH=HEAD) 2024-04-22T21:44:22.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:22 smithi188 podman[43471]: 2024-04-22 21:44:22.649250641 +0000 UTC m=+0.997496757 container cleanup 1aa8711a0f5e7dec13b36be36f3270e20bb63c8489927488977193a98074f41d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-b, vcs-type=git, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, name=centos-stream, build-date=2022-05-03T08:36:31.336870, ceph=True, architecture=x86_64, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T21:44:22.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:22 smithi188 bash[43471]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon-b 2024-04-22T21:44:22.756 INFO:journalctl@ceph.mon.b.smithi188.stdout:Apr 22 21:44:22 smithi188 bash[43497]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mon.b" found: no such container 2024-04-22T21:44:22.903 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mon.b.service' 2024-04-22T21:44:22.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:44:22.951 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-22T21:44:22.951 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-22T21:44:22.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mgr.y 2024-04-22T21:44:23.306 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:44:23 smithi165 systemd[1]: Stopping Ceph mgr.y for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:23.306 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:44:23 smithi165 bash[54544]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr.y" found: no such container 2024-04-22T21:44:23.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:44:23 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y[22063]: 2024-04-22T21:44:23.186+0000 7f5be0e24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:44:23.307 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:44:23 smithi165 podman[54557]: 2024-04-22 21:44:23.217115479 +0000 UTC m=+0.162198245 container died edb11b64e074c5af1150a192f5c34e49a78cf7074b3a611a6f4e322572fef248 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, ceph=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T21:44:23.806 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:44:23 smithi165 podman[54557]: 2024-04-22 21:44:23.702865196 +0000 UTC m=+0.647947970 container cleanup edb11b64e074c5af1150a192f5c34e49a78cf7074b3a611a6f4e322572fef248 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., architecture=x86_64, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, name=centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T21:44:23.806 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:44:23 smithi165 bash[54557]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-y 2024-04-22T21:44:23.807 INFO:journalctl@ceph.mgr.y.smithi165.stdout:Apr 22 21:44:23 smithi165 bash[54584]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr.y" found: no such container 2024-04-22T21:44:23.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-22T21:44:24.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:44:24.012 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-22T21:44:24.012 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-22T21:44:24.012 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mgr.x 2024-04-22T21:44:24.506 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:44:24 smithi188 systemd[1]: Stopping Ceph mgr.x for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:24.506 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:44:24 smithi188 bash[43551]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr.x" found: no such container 2024-04-22T21:44:24.506 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:44:24 smithi188 podman[43564]: 2024-04-22 21:44:24.258022058 +0000 UTC m=+0.151505541 container died 9f10aded29b4f16c1c80bf1f2bf33c2b5f976a37f127e02e7d1c6f5d022cff8c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vcs-type=git, version=8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-22T21:44:24.885 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:44:24 smithi188 podman[43564]: 2024-04-22 21:44:24.645309873 +0000 UTC m=+0.538793368 container cleanup 9f10aded29b4f16c1c80bf1f2bf33c2b5f976a37f127e02e7d1c6f5d022cff8c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, ceph=True, GIT_CLEAN=True, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., architecture=x86_64, distribution-scope=public, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD) 2024-04-22T21:44:24.885 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:44:24 smithi188 bash[43564]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr-x 2024-04-22T21:44:24.885 INFO:journalctl@ceph.mgr.x.smithi188.stdout:Apr 22 21:44:24 smithi188 bash[43591]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-mgr.x" found: no such container 2024-04-22T21:44:24.889 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-22T21:44:24.922 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:44:24.923 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-22T21:44:24.923 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T21:44:24.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.0 2024-04-22T21:44:25.209 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:24 smithi165 systemd[1]: Stopping Ceph osd.0 for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:25.209 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:25 smithi165 bash[54638]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-22T21:44:25.209 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:25 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0[29420]: 2024-04-22T21:44:25.132+0000 7f611a309700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:44:25.209 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:25 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0[29420]: 2024-04-22T21:44:25.132+0000 7f611a309700 -1 osd.0 96 *** Got signal Terminated *** 2024-04-22T21:44:25.210 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:25 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0[29420]: 2024-04-22T21:44:25.132+0000 7f611a309700 -1 osd.0 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:44:30.806 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:30 smithi165 podman[54649]: 2024-04-22 21:44:30.472132371 +0000 UTC m=+5.446057843 container died b414e5f11e45737c09078e9f85312195cbd70b05c4f2dae3b7258f8d04520a08 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, ceph=True, architecture=x86_64, io.openshift.expose-services=, version=8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream) 2024-04-22T21:44:31.298 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:31 smithi165 podman[54649]: 2024-04-22 21:44:31.106970194 +0000 UTC m=+6.080895666 container cleanup b414e5f11e45737c09078e9f85312195cbd70b05c4f2dae3b7258f8d04520a08 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, vcs-type=git, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, release=754, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., architecture=x86_64) 2024-04-22T21:44:31.298 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:31 smithi165 bash[54649]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0 2024-04-22T21:44:31.556 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:31 smithi165 podman[54678]: 2024-04-22 21:44:31.316528172 +0000 UTC m=+0.841974346 container remove b414e5f11e45737c09078e9f85312195cbd70b05c4f2dae3b7258f8d04520a08 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, name=centos-stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git) 2024-04-22T21:44:31.556 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:31 smithi165 bash[54690]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-22T21:44:31.841 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:31 smithi165 podman[54757]: 2024-04-22 21:44:31.506490798 +0000 UTC m=+0.019054521 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:44:31.841 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:31 smithi165 podman[54757]: 2024-04-22 21:44:31.617181469 +0000 UTC m=+0.129745232 container create 4961ce3a93356a230fa227d3e2c024820dcf260462898f8cb17404aa9386498e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, name=centos-stream, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.openshift.expose-services=, release=754, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD) 2024-04-22T21:44:31.841 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:31 smithi165 podman[54757]: 2024-04-22 21:44:31.861040444 +0000 UTC m=+0.373604148 container init 4961ce3a93356a230fa227d3e2c024820dcf260462898f8cb17404aa9386498e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.19.8, ceph=True, maintainer=Guillaume Abrioux , vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, architecture=x86_64, GIT_BRANCH=HEAD, name=centos-stream, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8) 2024-04-22T21:44:32.138 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:31 smithi165 podman[54757]: 2024-04-22 21:44:31.864403991 +0000 UTC m=+0.376967698 container start 4961ce3a93356a230fa227d3e2c024820dcf260462898f8cb17404aa9386498e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_CLEAN=True, name=centos-stream, release=754, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, RELEASE=HEAD, io.openshift.tags=base centos centos-stream) 2024-04-22T21:44:32.138 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:31 smithi165 podman[54757]: 2024-04-22 21:44:31.969591832 +0000 UTC m=+0.482155551 container attach 4961ce3a93356a230fa227d3e2c024820dcf260462898f8cb17404aa9386498e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , distribution-scope=public, release=754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, architecture=x86_64, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, version=8) 2024-04-22T21:44:32.461 INFO:journalctl@ceph.osd.0.smithi165.stdout:Apr 22 21:44:32 smithi165 podman[54757]: 2024-04-22 21:44:32.158011985 +0000 UTC m=+0.670575691 container died 4961ce3a93356a230fa227d3e2c024820dcf260462898f8cb17404aa9386498e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-0-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, release=754, version=8, com.redhat.component=centos-stream-container, ceph=True) 2024-04-22T21:44:32.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T21:44:32.896 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:44:32.896 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T21:44:32.896 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T21:44:32.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.1 2024-04-22T21:44:33.114 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:32 smithi165 systemd[1]: Stopping Ceph osd.1 for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:33.114 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:33 smithi165 bash[54835]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-22T21:44:33.557 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1[32379]: 2024-04-22T21:44:33.134+0000 7f1c51df5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:44:33.558 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1[32379]: 2024-04-22T21:44:33.134+0000 7f1c51df5700 -1 osd.1 96 *** Got signal Terminated *** 2024-04-22T21:44:33.558 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:33 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1[32379]: 2024-04-22T21:44:33.134+0000 7f1c51df5700 -1 osd.1 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:44:38.731 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:38 smithi165 podman[54848]: 2024-04-22 21:44:38.441876882 +0000 UTC m=+5.414368161 container died 12a0b8284d72b32738a27ad62357d9887c73035d60003121a5be1c6f0e0b052f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, release=754, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, ceph=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, name=centos-stream, io.openshift.expose-services=) 2024-04-22T21:44:39.056 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:38 smithi165 podman[54848]: 2024-04-22 21:44:38.87528125 +0000 UTC m=+5.847772527 container cleanup 12a0b8284d72b32738a27ad62357d9887c73035d60003121a5be1c6f0e0b052f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.19.8, GIT_CLEAN=True, distribution-scope=public, vcs-type=git, release=754, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, name=centos-stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T21:44:39.057 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:38 smithi165 bash[54848]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1 2024-04-22T21:44:39.347 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:39 smithi165 podman[54868]: 2024-04-22 21:44:39.084955657 +0000 UTC m=+0.643154365 container remove 12a0b8284d72b32738a27ad62357d9887c73035d60003121a5be1c6f0e0b052f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.buildah.version=1.19.8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, version=8, ceph=True, architecture=x86_64, distribution-scope=public, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD) 2024-04-22T21:44:39.347 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:39 smithi165 bash[54882]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-22T21:44:39.601 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:39 smithi165 podman[54943]: 2024-04-22 21:44:39.271128449 +0000 UTC m=+0.018941066 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:44:39.601 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:39 smithi165 podman[54943]: 2024-04-22 21:44:39.410762039 +0000 UTC m=+0.158574660 container create 4a287d4afc4fcba46308d5ba51904fef07f65cd9865dcaeb9beb234415466956 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., release=754, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, version=8, maintainer=Guillaume Abrioux , ceph=True, io.openshift.expose-services=, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8) 2024-04-22T21:44:39.601 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:39 smithi165 podman[54943]: 2024-04-22 21:44:39.621126051 +0000 UTC m=+0.368938669 container init 4a287d4afc4fcba46308d5ba51904fef07f65cd9865dcaeb9beb234415466956 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.19.8, version=8, vendor=Red Hat, Inc., vcs-type=git, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, name=centos-stream, ceph=True) 2024-04-22T21:44:39.898 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:39 smithi165 podman[54943]: 2024-04-22 21:44:39.624409302 +0000 UTC m=+0.372221923 container start 4a287d4afc4fcba46308d5ba51904fef07f65cd9865dcaeb9beb234415466956 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, build-date=2022-05-03T08:36:31.336870, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, distribution-scope=public, ceph=True, io.openshift.expose-services=, version=8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_CLEAN=True, architecture=x86_64, RELEASE=HEAD) 2024-04-22T21:44:39.898 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:39 smithi165 podman[54943]: 2024-04-22 21:44:39.704522933 +0000 UTC m=+0.452335555 container attach 4a287d4afc4fcba46308d5ba51904fef07f65cd9865dcaeb9beb234415466956 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, release=754, distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.19.8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container) 2024-04-22T21:44:40.195 INFO:journalctl@ceph.osd.1.smithi165.stdout:Apr 22 21:44:39 smithi165 podman[54943]: 2024-04-22 21:44:39.917978968 +0000 UTC m=+0.665791585 container died 4a287d4afc4fcba46308d5ba51904fef07f65cd9865dcaeb9beb234415466956 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-1-deactivate, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, version=8, distribution-scope=public, ceph=True, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T21:44:40.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T21:44:40.572 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:44:40.572 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T21:44:40.573 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T21:44:40.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.2 2024-04-22T21:44:40.768 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:40 smithi165 systemd[1]: Stopping Ceph osd.2 for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:40.768 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:40 smithi165 bash[55023]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-22T21:44:41.056 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:40 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2[35305]: 2024-04-22T21:44:40.794+0000 7f8607813700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:44:41.057 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:40 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2[35305]: 2024-04-22T21:44:40.794+0000 7f8607813700 -1 osd.2 96 *** Got signal Terminated *** 2024-04-22T21:44:41.057 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:40 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2[35305]: 2024-04-22T21:44:40.794+0000 7f8607813700 -1 osd.2 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:44:46.557 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:46 smithi165 podman[55036]: 2024-04-22 21:44:46.267322136 +0000 UTC m=+5.553404772 container died 84c9568b476f309bcebb2f1af267ef89eb22a55e9f9ccde65eaa0d17facbffb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , name=centos-stream, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-22T21:44:47.057 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:46 smithi165 podman[55036]: 2024-04-22 21:44:46.710125427 +0000 UTC m=+5.996208062 container cleanup 84c9568b476f309bcebb2f1af267ef89eb22a55e9f9ccde65eaa0d17facbffb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2, release=754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, ceph=True, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, version=8) 2024-04-22T21:44:47.057 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:46 smithi165 bash[55036]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2 2024-04-22T21:44:47.057 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:46 smithi165 podman[55064]: 2024-04-22 21:44:46.911546476 +0000 UTC m=+0.646017283 container remove 84c9568b476f309bcebb2f1af267ef89eb22a55e9f9ccde65eaa0d17facbffb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, release=754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-22T21:44:47.057 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:46 smithi165 bash[55078]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-22T21:44:47.444 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:47 smithi165 podman[55140]: 2024-04-22 21:44:47.053233316 +0000 UTC m=+0.018730705 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:44:47.444 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:47 smithi165 podman[55140]: 2024-04-22 21:44:47.178743275 +0000 UTC m=+0.144240650 container create 659ad6bd8b0e4aeeaf0794e4cd2a5262dce24234420c58cd91c5e14ddd43d209 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, name=centos-stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, release=754, version=8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.openshift.expose-services=) 2024-04-22T21:44:47.444 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:47 smithi165 podman[55140]: 2024-04-22 21:44:47.389157559 +0000 UTC m=+0.354654934 container init 659ad6bd8b0e4aeeaf0794e4cd2a5262dce24234420c58cd91c5e14ddd43d209 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, distribution-scope=public, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, com.redhat.component=centos-stream-container, architecture=x86_64, io.openshift.expose-services=, release=754, GIT_BRANCH=HEAD, version=8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., ceph=True) 2024-04-22T21:44:47.444 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:47 smithi165 podman[55140]: 2024-04-22 21:44:47.392279983 +0000 UTC m=+0.357777359 container start 659ad6bd8b0e4aeeaf0794e4cd2a5262dce24234420c58cd91c5e14ddd43d209 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, RELEASE=HEAD, architecture=x86_64, version=8, io.openshift.tags=base centos centos-stream, ceph=True, io.openshift.expose-services=, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T21:44:47.444 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:47 smithi165 podman[55140]: 2024-04-22 21:44:47.463927477 +0000 UTC m=+0.429424861 container attach 659ad6bd8b0e4aeeaf0794e4cd2a5262dce24234420c58cd91c5e14ddd43d209 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, release=754, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T21:44:47.806 INFO:journalctl@ceph.osd.2.smithi165.stdout:Apr 22 21:44:47 smithi165 podman[55140]: 2024-04-22 21:44:47.670600384 +0000 UTC m=+0.636097770 container died 659ad6bd8b0e4aeeaf0794e4cd2a5262dce24234420c58cd91c5e14ddd43d209 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-2-deactivate, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, vcs-type=git, name=centos-stream, RELEASE=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T21:44:48.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T21:44:48.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:44:48.363 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T21:44:48.363 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T21:44:48.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.3 2024-04-22T21:44:48.556 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:48 smithi165 systemd[1]: Stopping Ceph osd.3 for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:48.556 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:48 smithi165 bash[55217]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-22T21:44:49.056 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:48 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3[38286]: 2024-04-22T21:44:48.619+0000 7f2ce97ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:44:49.056 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:48 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3[38286]: 2024-04-22T21:44:48.619+0000 7f2ce97ea700 -1 osd.3 96 *** Got signal Terminated *** 2024-04-22T21:44:49.056 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:48 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3[38286]: 2024-04-22T21:44:48.619+0000 7f2ce97ea700 -1 osd.3 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:44:51.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:44:51 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:44:51.496+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:44:52.056 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:51 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3[38286]: 2024-04-22T21:44:51.944+0000 7f2ce5183700 -1 osd.3 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:28.213500+0000 front 2024-04-22T21:44:28.213398+0000 (oldest deadline 2024-04-22T21:44:51.713559+0000) 2024-04-22T21:44:52.755 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:44:52 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:44:52.470+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:44:53.306 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:52 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3[38286]: 2024-04-22T21:44:52.958+0000 7f2ce5183700 -1 osd.3 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:28.213500+0000 front 2024-04-22T21:44:28.213398+0000 (oldest deadline 2024-04-22T21:44:51.713559+0000) 2024-04-22T21:44:53.638 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:44:53 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:44:53.390+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:44:53.638 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:44:53 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:44:53.460+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:44:54.006 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:44:53 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:44:53.655+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:44:54.350 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:54 smithi165 podman[55230]: 2024-04-22 21:44:54.096524617 +0000 UTC m=+5.601290062 container died 8ea698e06b17fef7798cd5d453418f5550989e1429e9306c1bef269fc0940a59 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870) 2024-04-22T21:44:54.708 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:54 smithi165 podman[55230]: 2024-04-22 21:44:54.544462389 +0000 UTC m=+6.049227842 container cleanup 8ea698e06b17fef7798cd5d453418f5550989e1429e9306c1bef269fc0940a59 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3, distribution-scope=public, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True) 2024-04-22T21:44:54.709 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:54 smithi165 bash[55230]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3 2024-04-22T21:44:54.756 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:54 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:44:54.579+0000 7f3ea8f92700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.809753+0000 front 2024-04-22T21:44:29.809839+0000 (oldest deadline 2024-04-22T21:44:53.909692+0000) 2024-04-22T21:44:54.756 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:44:54 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:44:54.650+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:44:54.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:44:54 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:44:54.421+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:44:54.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:44:54 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:44:54.497+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:44:54.978 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:54 smithi165 podman[55249]: 2024-04-22 21:44:54.72894173 +0000 UTC m=+0.626027538 container remove 8ea698e06b17fef7798cd5d453418f5550989e1429e9306c1bef269fc0940a59 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3, architecture=x86_64, GIT_CLEAN=True, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, version=8, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T21:44:54.978 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:54 smithi165 bash[55262]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-22T21:44:55.236 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:54 smithi165 podman[55325]: 2024-04-22 21:44:54.901246754 +0000 UTC m=+0.019163832 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:44:55.236 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:55 smithi165 podman[55325]: 2024-04-22 21:44:55.012751808 +0000 UTC m=+0.130668875 container create b6649c0e15e9c3f1b00e82cf2e733bd405310bfebc9f5d9374f230ac356e8a40 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, distribution-scope=public, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, architecture=x86_64, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, version=8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T21:44:55.237 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:55 smithi165 podman[55325]: 2024-04-22 21:44:55.256703029 +0000 UTC m=+0.374620103 container init b6649c0e15e9c3f1b00e82cf2e733bd405310bfebc9f5d9374f230ac356e8a40 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.openshift.expose-services=, version=8, release=754) 2024-04-22T21:44:55.523 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:55 smithi165 podman[55325]: 2024-04-22 21:44:55.25981717 +0000 UTC m=+0.377734239 container start b6649c0e15e9c3f1b00e82cf2e733bd405310bfebc9f5d9374f230ac356e8a40 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, RELEASE=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, vcs-type=git, build-date=2022-05-03T08:36:31.336870) 2024-04-22T21:44:55.524 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:55 smithi165 podman[55325]: 2024-04-22 21:44:55.348361221 +0000 UTC m=+0.466278300 container attach b6649c0e15e9c3f1b00e82cf2e733bd405310bfebc9f5d9374f230ac356e8a40 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, maintainer=Guillaume Abrioux , distribution-scope=public, vendor=Red Hat, Inc., ceph=True, RELEASE=HEAD, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T21:44:55.625 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:44:55.554+0000 7f3ea8f92700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.809753+0000 front 2024-04-22T21:44:29.809839+0000 (oldest deadline 2024-04-22T21:44:53.909692+0000) 2024-04-22T21:44:55.625 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:44:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:44:55.519+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:44:55.625 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:44:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:44:55.387+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:44:55.806 INFO:journalctl@ceph.osd.3.smithi165.stdout:Apr 22 21:44:55 smithi165 podman[55325]: 2024-04-22 21:44:55.544289241 +0000 UTC m=+0.662206307 container died b6649c0e15e9c3f1b00e82cf2e733bd405310bfebc9f5d9374f230ac356e8a40 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-3-deactivate, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, distribution-scope=public, io.openshift.tags=base centos centos-stream, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.component=centos-stream-container, vcs-type=git, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-22T21:44:56.005 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:44:55 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:44:55.644+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:44:56.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T21:44:56.234 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:44:56.234 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T21:44:56.235 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T21:44:56.235 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.4 2024-04-22T21:44:56.573 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:44:56 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:44:56.510+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:44:56.573 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:44:56 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:44:56.424+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:44:56.574 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:56 smithi188 systemd[1]: Stopping Ceph osd.4 for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:44:56.574 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:56 smithi188 bash[43661]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-22T21:44:56.574 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:56 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:44:56.461+0000 7f3eaddfa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:44:56.574 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:56 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:44:56.461+0000 7f3eaddfa700 -1 osd.4 96 *** Got signal Terminated *** 2024-04-22T21:44:56.574 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:56 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:44:56.461+0000 7f3eaddfa700 -1 osd.4 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:44:56.574 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:56 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:44:56.592+0000 7f3ea8f92700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.809753+0000 front 2024-04-22T21:44:29.809839+0000 (oldest deadline 2024-04-22T21:44:53.909692+0000) 2024-04-22T21:44:57.006 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:44:56 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:44:56.675+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:44:57.653 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:57 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:44:57.634+0000 7f3ea8f92700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.809753+0000 front 2024-04-22T21:44:29.809839+0000 (oldest deadline 2024-04-22T21:44:53.909692+0000) 2024-04-22T21:44:57.655 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:44:57 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:44:57.494+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:44:57.655 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:44:57 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:44:57.398+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:44:58.005 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:44:57 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:44:57.714+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:44:58.505 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:44:58 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:44:58.376+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:44:58.506 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:44:58 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:44:58.466+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:44:59.005 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:58 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:44:58.627+0000 7f3ea8f92700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.809753+0000 front 2024-04-22T21:44:29.809839+0000 (oldest deadline 2024-04-22T21:44:53.909692+0000) 2024-04-22T21:44:59.006 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:44:58 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:44:58.713+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:44:59.656 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:44:59 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:44:59.588+0000 7f3ea8f92700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.809753+0000 front 2024-04-22T21:44:29.809839+0000 (oldest deadline 2024-04-22T21:44:53.909692+0000) 2024-04-22T21:44:59.657 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:44:59 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:44:59.485+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:44:59.657 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:44:59 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:44:59.396+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:44:59.657 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:44:59 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:44:59.396+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:00.005 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:44:59 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:44:59.675+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:00.756 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:00 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4[26031]: 2024-04-22T21:45:00.628+0000 7f3ea8f92700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.809753+0000 front 2024-04-22T21:44:29.809839+0000 (oldest deadline 2024-04-22T21:44:53.909692+0000) 2024-04-22T21:45:00.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:00 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:00.465+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:00.757 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:00 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:00.657+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:00.757 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:00 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:00.408+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:00.757 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:00 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:00.408+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:01.756 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:01 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:01.619+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:01.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:01 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:01.444+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:01.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:01 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:01.444+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:01.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:01 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:01.457+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:01.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:01 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:01.457+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:02.193 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:01 smithi188 podman[43674]: 2024-04-22 21:45:01.902688136 +0000 UTC m=+5.564142468 container died a9f173f084dbfb979c31104bc875aa46c5fbea2b15c5d5de5156fcd840c85536 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4, io.openshift.expose-services=, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, RELEASE=HEAD, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.component=centos-stream-container, release=754, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux ) 2024-04-22T21:45:02.469 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:02 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:02.439+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:02.469 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:02 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:02.439+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:02.469 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:02 smithi188 podman[43674]: 2024-04-22 21:45:02.311602278 +0000 UTC m=+5.973056603 container cleanup a9f173f084dbfb979c31104bc875aa46c5fbea2b15c5d5de5156fcd840c85536 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4, architecture=x86_64, ceph=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, version=8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vcs-type=git, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T21:45:02.470 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:02 smithi188 bash[43674]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4 2024-04-22T21:45:02.470 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:02 smithi188 bash[43705]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-22T21:45:02.756 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:02 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:02.571+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:02.756 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:02 smithi188 podman[43691]: 2024-04-22 21:45:02.563141395 +0000 UTC m=+0.662631715 container remove a9f173f084dbfb979c31104bc875aa46c5fbea2b15c5d5de5156fcd840c85536 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, distribution-scope=public, release=754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, io.openshift.expose-services=, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, name=centos-stream, vcs-type=git) 2024-04-22T21:45:02.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:02 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:02.488+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:02.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:02 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:02.488+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:03.090 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:02 smithi188 podman[43772]: 2024-04-22 21:45:02.70580957 +0000 UTC m=+0.018981589 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:45:03.090 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:02 smithi188 podman[43772]: 2024-04-22 21:45:02.805545862 +0000 UTC m=+0.118717858 container create 6333ca324b7eeaf8f585b2b5fba3d0d5eab1cb6ba8a8b2e61b9c97eaf047dd0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T21:45:03.090 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:03 smithi188 podman[43772]: 2024-04-22 21:45:03.024847272 +0000 UTC m=+0.338019268 container init 6333ca324b7eeaf8f585b2b5fba3d0d5eab1cb6ba8a8b2e61b9c97eaf047dd0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870) 2024-04-22T21:45:03.090 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:03 smithi188 podman[43772]: 2024-04-22 21:45:03.028050378 +0000 UTC m=+0.341222378 container start 6333ca324b7eeaf8f585b2b5fba3d0d5eab1cb6ba8a8b2e61b9c97eaf047dd0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.expose-services=, distribution-scope=public, GIT_BRANCH=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., name=centos-stream, RELEASE=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, ceph=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T21:45:03.090 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:03 smithi188 podman[43772]: 2024-04-22 21:45:03.10979948 +0000 UTC m=+0.422971479 container attach 6333ca324b7eeaf8f585b2b5fba3d0d5eab1cb6ba8a8b2e61b9c97eaf047dd0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, version=8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.component=centos-stream-container, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, name=centos-stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8) 2024-04-22T21:45:03.394 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 22 21:45:03 smithi188 podman[43772]: 2024-04-22 21:45:03.314831727 +0000 UTC m=+0.628003746 container died 6333ca324b7eeaf8f585b2b5fba3d0d5eab1cb6ba8a8b2e61b9c97eaf047dd0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vcs-type=git, ceph=True, io.openshift.expose-services=, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , architecture=x86_64, RELEASE=HEAD, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T21:45:03.659 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:03 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:03.591+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:03.660 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:03 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:03.531+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:03.660 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:03 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:03.531+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:03.660 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:03 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:03.413+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:03.660 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:03 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:03.413+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:03.965 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T21:45:04.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:45:04.006 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T21:45:04.006 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T21:45:04.006 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.5 2024-04-22T21:45:04.225 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:04 smithi188 systemd[1]: Stopping Ceph osd.5 for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:45:04.225 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:04 smithi188 bash[43850]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-22T21:45:04.504 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:04 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:04.249+0000 7f1315efb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:45:04.505 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:04 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:04.249+0000 7f1315efb700 -1 osd.5 96 *** Got signal Terminated *** 2024-04-22T21:45:04.505 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:04 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:04.249+0000 7f1315efb700 -1 osd.5 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:45:04.505 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:04 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:04.421+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:04.505 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:04 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:04.421+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:04.755 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:04 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:04.639+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:04.756 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:04 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:04.639+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:33.731751+0000 front 2024-04-22T21:44:33.731789+0000 (oldest deadline 2024-04-22T21:45:03.732086+0000) 2024-04-22T21:45:04.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:04 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:04.523+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:04.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:04 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:04.523+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:05.756 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:05.602+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:05.756 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:05.602+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:33.731751+0000 front 2024-04-22T21:44:33.731789+0000 (oldest deadline 2024-04-22T21:45:03.732086+0000) 2024-04-22T21:45:05.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:05.463+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:05.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:05.463+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:05.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:05.515+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:05.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:05 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:05.515+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:06.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:06.467+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:06.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:06.467+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:06.757 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:06.570+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:06.757 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:06.570+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:33.731751+0000 front 2024-04-22T21:44:33.731789+0000 (oldest deadline 2024-04-22T21:45:03.732086+0000) 2024-04-22T21:45:06.758 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:06.510+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:06.758 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:06 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:06.510+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:07.758 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:07 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:07.490+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:07.759 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:07 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:07.490+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:07.759 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:07 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:07.580+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:07.760 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:07 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:07.580+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:33.731751+0000 front 2024-04-22T21:44:33.731789+0000 (oldest deadline 2024-04-22T21:45:03.732086+0000) 2024-04-22T21:45:07.761 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:07 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:07.560+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:07.761 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:07 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:07.560+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:08.755 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:08.476+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:08.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:08.476+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:08.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:08.476+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:43.769224+0000 front 2024-04-22T21:44:43.769235+0000 (oldest deadline 2024-04-22T21:45:08.469272+0000) 2024-04-22T21:45:08.756 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:08.557+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.730673+0000 front 2024-04-22T21:44:29.730782+0000 (oldest deadline 2024-04-22T21:44:52.630673+0000) 2024-04-22T21:45:08.756 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5[28919]: 2024-04-22T21:45:08.557+0000 7f1313898700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:33.731751+0000 front 2024-04-22T21:44:33.731789+0000 (oldest deadline 2024-04-22T21:45:03.732086+0000) 2024-04-22T21:45:08.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:08.555+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:08.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:08 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:08.555+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:09.756 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:09 smithi188 podman[43863]: 2024-04-22 21:45:09.59269821 +0000 UTC m=+5.449365087 container died a56baa2e0a108a4f776a9974bb196f5a8083bced3625a4979169f7e46ef64eec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., version=8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, ceph=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T21:45:09.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:09.456+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:09.757 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:09.456+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:09.757 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:09.456+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:43.769224+0000 front 2024-04-22T21:44:43.769235+0000 (oldest deadline 2024-04-22T21:45:08.469272+0000) 2024-04-22T21:45:09.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:09.531+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:09.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:09.531+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:09.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:09 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:09.531+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:10.255 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:10 smithi188 podman[43863]: 2024-04-22 21:45:10.05536774 +0000 UTC m=+5.912034658 container cleanup a56baa2e0a108a4f776a9974bb196f5a8083bced3625a4979169f7e46ef64eec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5, vcs-type=git, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, vendor=Red Hat, Inc., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, version=8, RELEASE=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T21:45:10.256 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:10 smithi188 bash[43863]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5 2024-04-22T21:45:10.256 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:10 smithi188 bash[43907]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-22T21:45:10.642 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:10 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:10.503+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:10.642 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:10 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:10.503+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:10.642 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:10 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:10.503+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:43.769224+0000 front 2024-04-22T21:44:43.769235+0000 (oldest deadline 2024-04-22T21:45:08.469272+0000) 2024-04-22T21:45:10.643 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:10 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:10.509+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:10.643 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:10 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:10.509+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:10.643 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:10 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:10.509+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:10.643 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:10 smithi188 podman[43891]: 2024-04-22 21:45:10.356577513 +0000 UTC m=+0.766102221 container remove a56baa2e0a108a4f776a9974bb196f5a8083bced3625a4979169f7e46ef64eec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, ceph=True, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vcs-type=git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True) 2024-04-22T21:45:10.643 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:10 smithi188 podman[43975]: 2024-04-22 21:45:10.460402088 +0000 UTC m=+0.019185801 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:45:10.644 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:10 smithi188 podman[43975]: 2024-04-22 21:45:10.565598563 +0000 UTC m=+0.124382276 container create 3f88612b18e50b72da7aa1041336f234f3afd4d8baae2221ae35ee1550fb28d2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, name=centos-stream, com.redhat.component=centos-stream-container, GIT_CLEAN=True, version=8, vendor=Red Hat, Inc., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public) 2024-04-22T21:45:11.006 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:10 smithi188 podman[43975]: 2024-04-22 21:45:10.768002085 +0000 UTC m=+0.326785797 container init 3f88612b18e50b72da7aa1041336f234f3afd4d8baae2221ae35ee1550fb28d2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8) 2024-04-22T21:45:11.006 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:10 smithi188 podman[43975]: 2024-04-22 21:45:10.771029476 +0000 UTC m=+0.329813179 container start 3f88612b18e50b72da7aa1041336f234f3afd4d8baae2221ae35ee1550fb28d2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, RELEASE=HEAD, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, release=754, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, version=8, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-type=git, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T21:45:11.006 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:10 smithi188 podman[43975]: 2024-04-22 21:45:10.851263951 +0000 UTC m=+0.410047664 container attach 3f88612b18e50b72da7aa1041336f234f3afd4d8baae2221ae35ee1550fb28d2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, release=754, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.19.8, vcs-type=git) 2024-04-22T21:45:11.333 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 22 21:45:11 smithi188 podman[43975]: 2024-04-22 21:45:11.060982862 +0000 UTC m=+0.619766575 container died 3f88612b18e50b72da7aa1041336f234f3afd4d8baae2221ae35ee1550fb28d2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-5-deactivate, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., ceph=True, name=centos-stream, RELEASE=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, version=8, release=754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public) 2024-04-22T21:45:11.609 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:11 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:11.522+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:11.609 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:11 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:11.522+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:11.609 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:11 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:11.522+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:11.610 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:11 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:11.520+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:11.610 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:11 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:11.520+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:11.610 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:11 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:11.520+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:43.769224+0000 front 2024-04-22T21:44:43.769235+0000 (oldest deadline 2024-04-22T21:45:08.469272+0000) 2024-04-22T21:45:11.635 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T21:45:11.674 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:45:11.674 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T21:45:11.675 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T21:45:11.675 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.6 2024-04-22T21:45:11.887 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:11 smithi188 systemd[1]: Stopping Ceph osd.6 for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:45:11.888 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:11 smithi188 bash[44052]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-22T21:45:11.888 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:11 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:11.907+0000 7fa5e9a82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:45:11.888 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:11 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:11.907+0000 7fa5e9a82700 -1 osd.6 96 *** Got signal Terminated *** 2024-04-22T21:45:11.888 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:11 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:11.907+0000 7fa5e9a82700 -1 osd.6 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:45:12.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:12 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:12.527+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:12.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:12 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:12.527+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:12.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:12 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:12.527+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:12.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:12 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:12.478+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:12.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:12 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:12.478+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:12.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:12 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:12.478+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:43.769224+0000 front 2024-04-22T21:44:43.769235+0000 (oldest deadline 2024-04-22T21:45:08.469272+0000) 2024-04-22T21:45:13.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:13 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:13.506+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:13.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:13 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:13.506+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:13.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:13 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:13.506+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:43.769224+0000 front 2024-04-22T21:44:43.769235+0000 (oldest deadline 2024-04-22T21:45:08.469272+0000) 2024-04-22T21:45:13.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:13 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:13.501+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:13.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:13 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:13.501+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:13.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:13 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:13.501+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:14.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:14.485+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:14.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:14.485+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:14.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:14.485+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:43.769224+0000 front 2024-04-22T21:44:43.769235+0000 (oldest deadline 2024-04-22T21:45:08.469272+0000) 2024-04-22T21:45:14.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:14.498+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:14.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:14.498+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:14.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:14.498+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:14.757 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:14 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:14.498+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:15.506 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:15.447+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:15.506 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:15.447+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:15.506 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:15.447+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:43.769224+0000 front 2024-04-22T21:44:43.769235+0000 (oldest deadline 2024-04-22T21:45:08.469272+0000) 2024-04-22T21:45:16.005 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:15.526+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:16.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:15.526+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:16.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:15.526+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:16.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:15 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:15.526+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:16.755 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:16.525+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:16.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:16.525+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:16.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:16.525+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:16.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:16.525+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:16.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:16.403+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:27.967224+0000 front 2024-04-22T21:44:27.967142+0000 (oldest deadline 2024-04-22T21:44:52.667132+0000) 2024-04-22T21:45:16.756 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:16.403+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:37.968244+0000 front 2024-04-22T21:44:37.968206+0000 (oldest deadline 2024-04-22T21:44:58.468161+0000) 2024-04-22T21:45:16.757 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:16 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6[31804]: 2024-04-22T21:45:16.403+0000 7fa5e741f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:43.769224+0000 front 2024-04-22T21:44:43.769235+0000 (oldest deadline 2024-04-22T21:45:08.469272+0000) 2024-04-22T21:45:17.553 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:17.519+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:17.554 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:17.519+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:17.554 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:17.519+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:17.554 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:17 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:17.519+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:17.554 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:17 smithi188 podman[44065]: 2024-04-22 21:45:17.318046728 +0000 UTC m=+5.515617685 container died 1b040381578eb03a30f7effb1e50fc371179b9da4d9ffd3531ecaf0340e3c340 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, release=754, GIT_CLEAN=True, ceph=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T21:45:17.929 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:17 smithi188 podman[44065]: 2024-04-22 21:45:17.689813833 +0000 UTC m=+5.887384797 container cleanup 1b040381578eb03a30f7effb1e50fc371179b9da4d9ffd3531ecaf0340e3c340 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, architecture=x86_64, io.openshift.expose-services=, GIT_BRANCH=HEAD, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T21:45:17.929 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:17 smithi188 bash[44065]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6 2024-04-22T21:45:17.929 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:17 smithi188 bash[44097]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-22T21:45:18.221 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:17 smithi188 podman[44082]: 2024-04-22 21:45:17.949149889 +0000 UTC m=+0.629062568 container remove 1b040381578eb03a30f7effb1e50fc371179b9da4d9ffd3531ecaf0340e3c340 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, vendor=Red Hat, Inc., version=8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, name=centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, architecture=x86_64, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T21:45:18.221 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:18 smithi188 podman[44162]: 2024-04-22 21:45:18.133137199 +0000 UTC m=+0.115700110 container create 8ec23201c217394301043c6c67801b0602bda279f385aaede9af01d892edf9c9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, name=centos-stream, io.buildah.version=1.19.8, vcs-type=git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.openshift.tags=base centos centos-stream, ceph=True, GIT_CLEAN=True, vendor=Red Hat, Inc., distribution-scope=public, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T21:45:18.221 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:18 smithi188 podman[44162]: 2024-04-22 21:45:18.036653413 +0000 UTC m=+0.019216346 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:45:18.505 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:18 smithi188 podman[44162]: 2024-04-22 21:45:18.352128137 +0000 UTC m=+0.334691048 container init 8ec23201c217394301043c6c67801b0602bda279f385aaede9af01d892edf9c9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, version=8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T21:45:18.506 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:18 smithi188 podman[44162]: 2024-04-22 21:45:18.355303727 +0000 UTC m=+0.337866646 container start 8ec23201c217394301043c6c67801b0602bda279f385aaede9af01d892edf9c9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, com.redhat.component=centos-stream-container, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, maintainer=Guillaume Abrioux , architecture=x86_64, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_BRANCH=HEAD) 2024-04-22T21:45:18.506 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:18 smithi188 podman[44162]: 2024-04-22 21:45:18.435349877 +0000 UTC m=+0.417912799 container attach 8ec23201c217394301043c6c67801b0602bda279f385aaede9af01d892edf9c9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD) 2024-04-22T21:45:18.909 INFO:journalctl@ceph.osd.6.smithi188.stdout:Apr 22 21:45:18 smithi188 podman[44162]: 2024-04-22 21:45:18.634720041 +0000 UTC m=+0.617282953 container died 8ec23201c217394301043c6c67801b0602bda279f385aaede9af01d892edf9c9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-6-deactivate, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, release=754, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, version=8, io.k8s.display-name=CentOS Stream 8) 2024-04-22T21:45:18.910 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:18 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:18.554+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:18.910 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:18 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:18.554+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:18.910 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:18 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:18.554+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:18.910 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:18 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:18.554+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:19.183 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T21:45:19.220 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:45:19.220 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T21:45:19.221 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T21:45:19.221 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.7 2024-04-22T21:45:19.438 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:19 smithi188 systemd[1]: Stopping Ceph osd.7 for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:45:19.439 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:19 smithi188 bash[44241]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-22T21:45:19.439 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:19 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:19.458+0000 7f3cd517a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T21:45:19.755 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:19 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:19.458+0000 7f3cd517a700 -1 osd.7 96 *** Got signal Terminated *** 2024-04-22T21:45:19.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:19 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:19.458+0000 7f3cd517a700 -1 osd.7 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T21:45:19.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:19 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:19.594+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:19.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:19 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:19.594+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:19.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:19 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:19.594+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:19.756 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:19 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:19.594+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:21.005 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:20 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:20.623+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:21.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:20 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:20.623+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:21.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:20 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:20.623+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:21.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:20 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:20.623+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:22.005 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:21 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:21.594+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:22.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:21 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:21.594+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:22.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:21 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:21.594+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:22.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:21 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:21.594+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:23.005 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:22 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:22.625+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:23.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:22 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:22.625+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:23.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:22 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:22.625+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:23.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:22 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:22.625+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:23.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:22 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:22.625+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.188:6804 osd.4 since back 2024-04-22T21:44:58.476555+0000 front 2024-04-22T21:44:58.476573+0000 (oldest deadline 2024-04-22T21:45:21.976521+0000) 2024-04-22T21:45:24.005 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:23.613+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6806 osd.0 since back 2024-04-22T21:44:29.472203+0000 front 2024-04-22T21:44:29.472265+0000 (oldest deadline 2024-04-22T21:44:50.572175+0000) 2024-04-22T21:45:24.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:23.613+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6814 osd.1 since back 2024-04-22T21:44:36.373830+0000 front 2024-04-22T21:44:36.373772+0000 (oldest deadline 2024-04-22T21:45:01.073682+0000) 2024-04-22T21:45:24.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:23.613+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6822 osd.2 since back 2024-04-22T21:44:45.075057+0000 front 2024-04-22T21:44:45.075050+0000 (oldest deadline 2024-04-22T21:45:08.574911+0000) 2024-04-22T21:45:24.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:23.613+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.165:6830 osd.3 since back 2024-04-22T21:44:51.475830+0000 front 2024-04-22T21:44:51.475788+0000 (oldest deadline 2024-04-22T21:45:14.375779+0000) 2024-04-22T21:45:24.006 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:23 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7[34784]: 2024-04-22T21:45:23.613+0000 7f3cce30e700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.188:6804 osd.4 since back 2024-04-22T21:44:58.476555+0000 front 2024-04-22T21:44:58.476573+0000 (oldest deadline 2024-04-22T21:45:21.976521+0000) 2024-04-22T21:45:25.080 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:24 smithi188 podman[44255]: 2024-04-22 21:45:24.822846964 +0000 UTC m=+5.462631909 container died 41502d6e86988a544de5df4ad3f3b1645725072421df1f5c14264909ec2fb36f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-type=git, io.buildah.version=1.19.8, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.openshift.expose-services=, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T21:45:25.448 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:25 smithi188 podman[44255]: 2024-04-22 21:45:25.266437931 +0000 UTC m=+5.906222904 container cleanup 41502d6e86988a544de5df4ad3f3b1645725072421df1f5c14264909ec2fb36f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7, architecture=x86_64, version=8, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vcs-type=git, RELEASE=HEAD, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T21:45:25.448 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:25 smithi188 bash[44255]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7 2024-04-22T21:45:25.698 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:25 smithi188 podman[44282]: 2024-04-22 21:45:25.468019977 +0000 UTC m=+0.643647650 container remove 41502d6e86988a544de5df4ad3f3b1645725072421df1f5c14264909ec2fb36f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, version=8, name=centos-stream, io.buildah.version=1.19.8, vcs-type=git, release=754, GIT_BRANCH=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T21:45:25.699 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:25 smithi188 bash[44295]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-22T21:45:25.699 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:25 smithi188 podman[44362]: 2024-04-22 21:45:25.611452118 +0000 UTC m=+0.018851720 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T21:45:25.699 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:25 smithi188 podman[44362]: 2024-04-22 21:45:25.718840692 +0000 UTC m=+0.126240325 container create 542f902fcf3b7568ee4243e36485946b32a7804f2a58d0637e2b45596225ffd0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, name=centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_CLEAN=True) 2024-04-22T21:45:25.984 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:25 smithi188 podman[44362]: 2024-04-22 21:45:25.929551112 +0000 UTC m=+0.336950700 container init 542f902fcf3b7568ee4243e36485946b32a7804f2a58d0637e2b45596225ffd0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, architecture=x86_64, vendor=Red Hat, Inc., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, ceph=True, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.expose-services=) 2024-04-22T21:45:25.984 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:25 smithi188 podman[44362]: 2024-04-22 21:45:25.93261038 +0000 UTC m=+0.340009967 container start 542f902fcf3b7568ee4243e36485946b32a7804f2a58d0637e2b45596225ffd0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_CLEAN=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T21:45:25.984 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:26 smithi188 podman[44362]: 2024-04-22 21:45:26.004268159 +0000 UTC m=+0.411667760 container attach 542f902fcf3b7568ee4243e36485946b32a7804f2a58d0637e2b45596225ffd0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, release=754, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_BRANCH=HEAD, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, vendor=Red Hat, Inc., version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T21:45:26.256 INFO:journalctl@ceph.osd.7.smithi188.stdout:Apr 22 21:45:26 smithi188 podman[44362]: 2024-04-22 21:45:26.217777967 +0000 UTC m=+0.625177546 container died 542f902fcf3b7568ee4243e36485946b32a7804f2a58d0637e2b45596225ffd0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-osd-7-deactivate, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, release=754, GIT_CLEAN=True, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., ceph=True, version=8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD) 2024-04-22T21:45:26.753 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T21:45:26.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:45:26.792 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T21:45:26.793 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-22T21:45:26.793 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@prometheus.a 2024-04-22T21:45:27.005 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:26 smithi188 systemd[1]: Stopping Ceph prometheus.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:45:27.006 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:26 smithi188 bash[44442]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-22T21:45:27.282 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-22T21:45:27.283 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-22T21:45:27.283 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-22T21:45:27.283 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-22T21:45:27.283 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-22T21:45:27.283 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-22T21:45:27.283 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-22T21:45:27.283 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-22T21:45:27.284 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-22T21:45:27.284 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-22T21:45:27.284 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-22T21:45:27.284 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a[39508]: ts=2024-04-22T21:45:27.028Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-22T21:45:27.284 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 podman[44455]: 2024-04-22 21:45:27.038385844 +0000 UTC m=+0.119740769 container died b1b3a948cedf3217709263558fa1c118ccdd2501898c6a5c30cdb33c810ba5bf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:45:27.642 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 podman[44455]: 2024-04-22 21:45:27.419555999 +0000 UTC m=+0.500910933 container cleanup b1b3a948cedf3217709263558fa1c118ccdd2501898c6a5c30cdb33c810ba5bf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T21:45:27.645 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 bash[44455]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus-a 2024-04-22T21:45:27.645 INFO:journalctl@ceph.prometheus.a.smithi188.stdout:Apr 22 21:45:27 smithi188 bash[44481]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-22T21:45:27.648 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@prometheus.a.service' 2024-04-22T21:45:27.683 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T21:45:27.683 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-22T21:45:27.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T21:45:29.358 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:29 smithi165 systemd[1]: Stopping Ceph node-exporter.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:45:29.358 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:29 smithi165 bash[55707]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter.a" found: no such container 2024-04-22T21:45:29.713 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:29 smithi165 podman[55720]: 2024-04-22 21:45:29.477613571 +0000 UTC m=+0.138692073 container died f4355268ef2fe03a9d2b730d57b32688880f04f5ab74ee7ed3a26d501c7e0192 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T21:45:30.056 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:29 smithi165 podman[55720]: 2024-04-22 21:45:29.92534632 +0000 UTC m=+0.586424830 container cleanup f4355268ef2fe03a9d2b730d57b32688880f04f5ab74ee7ed3a26d501c7e0192 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T21:45:30.056 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:29 smithi165 bash[55720]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a 2024-04-22T21:45:30.056 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:29 smithi165 bash[55748]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter.a" found: no such container 2024-04-22T21:45:30.509 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:30 smithi165 systemd[1]: Stopping Ceph alertmanager.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:45:30.509 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:30 smithi165 bash[55802]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-22T21:45:30.509 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:30 smithi165 podman[55735]: 2024-04-22 21:45:30.201692967 +0000 UTC m=+0.728381555 container remove f4355268ef2fe03a9d2b730d57b32688880f04f5ab74ee7ed3a26d501c7e0192 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T21:45:30.510 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:30 smithi165 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-22T21:45:30.510 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:30 smithi165 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-22T21:45:30.510 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:30 smithi165 systemd[1]: Stopped Ceph node-exporter.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:45:30.510 INFO:journalctl@ceph.node-exporter.a.smithi165.stdout:Apr 22 21:45:30 smithi165 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@node-exporter.a.service: Consumed 3.026s CPU time. 2024-04-22T21:45:30.806 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:30 smithi165 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a[46511]: level=info ts=2024-04-22T21:45:30.529Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-22T21:45:30.806 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:30 smithi165 podman[55816]: 2024-04-22 21:45:30.560061083 +0000 UTC m=+0.138947467 container died 5f88b3382f5836e2bfa79ed9d9a8aa07c922014885b57ec2616e2f42badbe9a7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T21:45:31.268 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:30 smithi165 podman[55816]: 2024-04-22 21:45:30.91286762 +0000 UTC m=+0.491754003 container cleanup 5f88b3382f5836e2bfa79ed9d9a8aa07c922014885b57ec2616e2f42badbe9a7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T21:45:31.268 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:30 smithi165 bash[55816]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a 2024-04-22T21:45:31.269 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:30 smithi165 bash[55844]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-22T21:45:31.546 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:31 smithi165 podman[55830]: 2024-04-22 21:45:31.288840456 +0000 UTC m=+0.726235675 container remove 5f88b3382f5836e2bfa79ed9d9a8aa07c922014885b57ec2616e2f42badbe9a7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T21:45:31.546 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:31 smithi165 podman[55830]: 2024-04-22 21:45:31.372913976 +0000 UTC m=+0.810309150 volume remove 74e8bfbca2e1aafbec07d9740dd6ef91164c94a168d2892a36d1402b7b6ee5df 2024-04-22T21:45:31.546 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:31 smithi165 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-22T21:45:31.546 INFO:journalctl@ceph.alertmanager.a.smithi165.stdout:Apr 22 21:45:31 smithi165 systemd[1]: Stopped Ceph alertmanager.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:46:05.303 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T21:46:07.005 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:06 smithi188 systemd[1]: Stopping Ceph node-exporter.b for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:46:07.006 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:06 smithi188 bash[44811]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter.b" found: no such container 2024-04-22T21:46:07.006 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:06 smithi188 podman[44824]: 2024-04-22 21:46:06.877825664 +0000 UTC m=+0.112989234 container died 211a953ad0ee2314903b007c3deff814eb0f168e890df47cd6adf2a56adfb654 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T21:46:07.300 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:07 smithi188 podman[44824]: 2024-04-22 21:46:07.277913281 +0000 UTC m=+0.513076852 container cleanup 211a953ad0ee2314903b007c3deff814eb0f168e890df47cd6adf2a56adfb654 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T21:46:07.301 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:07 smithi188 bash[44824]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b 2024-04-22T21:46:07.301 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:07 smithi188 bash[44851]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter.b" found: no such container 2024-04-22T21:46:07.729 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:07 smithi188 podman[44838]: 2024-04-22 21:46:07.554242872 +0000 UTC m=+0.676619279 container remove 211a953ad0ee2314903b007c3deff814eb0f168e890df47cd6adf2a56adfb654 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T21:46:07.730 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:07 smithi188 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-22T21:46:07.730 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:07 smithi188 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-22T21:46:07.730 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:07 smithi188 systemd[1]: Stopped Ceph node-exporter.b for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:46:07.730 INFO:journalctl@ceph.node-exporter.b.smithi188.stdout:Apr 22 21:46:07 smithi188 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@node-exporter.b.service: Consumed 3.061s CPU time. 2024-04-22T21:46:08.005 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:07 smithi188 systemd[1]: Stopping Ceph grafana.a for dccb09a4-00ef-11ef-bc93-c7b262605968... 2024-04-22T21:46:08.006 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:07 smithi188 bash[44919]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana.a" found: no such container 2024-04-22T21:46:08.006 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:07 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:46:07+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-22T21:46:08.006 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:07 smithi188 ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a[38716]: t=2024-04-22T21:46:07+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-04-22T21:46:08.006 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:07 smithi188 podman[44931]: 2024-04-22 21:46:07.919177481 +0000 UTC m=+0.130110869 container died 1184bc13bc40423f8cd8b88a30e37d23ac5f8eefbe93eb6a685bcc42fb8f6d5a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, architecture=x86_64, name=ubi8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, distribution-scope=public, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762) 2024-04-22T21:46:08.505 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:08 smithi188 podman[44931]: 2024-04-22 21:46:08.340284407 +0000 UTC m=+0.551217829 container cleanup 1184bc13bc40423f8cd8b88a30e37d23ac5f8eefbe93eb6a685bcc42fb8f6d5a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, release=236.1648460182, description=Ceph Grafana Container, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-22T21:46:08.506 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:08 smithi188 bash[44931]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a 2024-04-22T21:46:08.506 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:08 smithi188 bash[44960]: Error: no container with name or ID "ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana.a" found: no such container 2024-04-22T21:46:08.842 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:08 smithi188 podman[44945]: 2024-04-22 21:46:08.554212667 +0000 UTC m=+0.633387887 container remove 1184bc13bc40423f8cd8b88a30e37d23ac5f8eefbe93eb6a685bcc42fb8f6d5a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dccb09a4-00ef-11ef-bc93-c7b262605968-grafana-a, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.openshift.expose-services=, architecture=x86_64, version=8.5, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vendor=Red Hat, Inc.) 2024-04-22T21:46:08.842 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:08 smithi188 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-22T21:46:08.842 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:08 smithi188 systemd[1]: Stopped Ceph grafana.a for dccb09a4-00ef-11ef-bc93-c7b262605968. 2024-04-22T21:46:08.842 INFO:journalctl@ceph.grafana.a.smithi188.stdout:Apr 22 21:46:08 smithi188 systemd[1]: ceph-dccb09a4-00ef-11ef-bc93-c7b262605968@grafana.a.service: Consumed 2.710s CPU time. 2024-04-22T21:46:30.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:46:30.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T21:46:30.875 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T21:46:30.877 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763/remote/smithi165/crash 2024-04-22T21:46:30.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/crash -- . 2024-04-22T21:46:30.923 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T21:46:30.924 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now 2024-04-22T21:46:30.925 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763/remote/smithi188/crash 2024-04-22T21:46:30.927 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/crash -- . 2024-04-22T21:46:30.962 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T21:46:30.963 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2024-04-22T21:46:30.964 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T21:46:30.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-22T21:46:30.993 INFO:tasks.cephadm:Compressing logs... 2024-04-22T21:46:30.994 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T21:46:31.037 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T21:46:31.059 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T21:46:31.060 INFO:teuthology.orchestra.run.smithi188.stderr:: No such file or directory 2024-04-22T21:46:31.060 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.log 2024-04-22T21:46:31.060 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T21:46:31.060 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cephadm.log: gzip/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.log: -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T21:46:31.068 INFO:teuthology.orchestra.run.smithi165.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T21:46:31.068 INFO:teuthology.orchestra.run.smithi165.stderr:: No such file or directory 2024-04-22T21:46:31.068 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.log 2024-04-22T21:46:31.068 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T21:46:31.068 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T21:46:31.068 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi165.pbstvw.log 2024-04-22T21:46:31.068 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T21:46:31.069 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi165.pbstvw.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T21:46:31.069 INFO:teuthology.orchestra.run.smithi165.stderr: 62.5% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi165.pbstvw.log.gz 2024-04-22T21:46:31.069 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-22T21:46:31.069 INFO:teuthology.orchestra.run.smithi165.stderr: 92.8% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T21:46:31.069 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T21:46:31.069 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-22T21:46:31.069 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-22T21:46:31.070 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mon.c.log: 93.9% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T21:46:31.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T21:46:31.071 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T21:46:31.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-22T21:46:31.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.4.log: 87.4% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T21:46:31.071 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T21:46:31.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi188.ryxylz.log 2024-04-22T21:46:31.072 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip 90.2% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T21:46:31.072 INFO:teuthology.orchestra.run.smithi188.stderr: -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi188.wgqixq.log 2024-04-22T21:46:31.072 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T21:46:31.072 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi188.ryxylz.log: /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi188.wgqixq.log: 62.4% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi188.ryxylz.log.gz 2024-04-22T21:46:31.072 INFO:teuthology.orchestra.run.smithi188.stderr: 62.5% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi188.wgqixq.log.gz 2024-04-22T21:46:31.072 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T21:46:31.072 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log: 79.4% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T21:46:31.072 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-22T21:46:31.072 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mon.b.log: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T21:46:31.073 INFO:teuthology.orchestra.run.smithi188.stderr: 91.1% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-22T21:46:31.073 INFO:teuthology.orchestra.run.smithi188.stderr: 90.7% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T21:46:31.076 INFO:teuthology.orchestra.run.smithi165.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T21:46:31.076 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi165.hjvsmr.log 2024-04-22T21:46:31.077 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T21:46:31.077 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi165.hjvsmr.log: 62.6% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi165.hjvsmr.log.gz 2024-04-22T21:46:31.077 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/tcmu-runner.log 2024-04-22T21:46:31.077 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log: 88.2% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T21:46:31.077 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T21:46:31.077 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/tcmu-runner.log: 66.6% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/tcmu-runner.log.gz 2024-04-22T21:46:31.083 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T21:46:31.218 INFO:teuthology.orchestra.run.smithi188.stderr: 92.2% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-22T21:46:31.232 INFO:teuthology.orchestra.run.smithi165.stderr: 89.0% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-22T21:46:31.247 INFO:teuthology.orchestra.run.smithi165.stderr: 92.2% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-22T21:46:31.395 INFO:teuthology.orchestra.run.smithi165.stderr: 93.4% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T21:46:31.421 INFO:teuthology.orchestra.run.smithi188.stderr: 93.1% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T21:46:31.457 INFO:teuthology.orchestra.run.smithi188.stderr: 93.3% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T21:46:31.497 INFO:teuthology.orchestra.run.smithi188.stderr: 93.6% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T21:46:31.504 INFO:teuthology.orchestra.run.smithi188.stderr: 93.8% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T21:46:31.506 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-22T21:46:31.506 INFO:teuthology.orchestra.run.smithi188.stderr:real 0m0.455s 2024-04-22T21:46:31.507 INFO:teuthology.orchestra.run.smithi188.stderr:user 0m1.747s 2024-04-22T21:46:31.507 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m0.088s 2024-04-22T21:46:31.510 INFO:teuthology.orchestra.run.smithi165.stderr: 90.9% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-22T21:46:31.513 INFO:teuthology.orchestra.run.smithi165.stderr: 93.6% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T21:46:31.519 INFO:teuthology.orchestra.run.smithi165.stderr: 93.5% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T21:46:31.525 INFO:teuthology.orchestra.run.smithi165.stderr: 93.5% -- replaced with /var/log/ceph/dccb09a4-00ef-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T21:46:31.527 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-04-22T21:46:31.527 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m0.477s 2024-04-22T21:46:31.527 INFO:teuthology.orchestra.run.smithi165.stderr:user 0m2.435s 2024-04-22T21:46:31.527 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m0.107s 2024-04-22T21:46:31.528 INFO:tasks.cephadm:Archiving logs... 2024-04-22T21:46:31.529 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763/remote/smithi165/log 2024-04-22T21:46:31.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T21:46:31.916 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763/remote/smithi188/log 2024-04-22T21:46:31.917 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T21:46:32.205 INFO:tasks.cephadm:Removing cluster... 2024-04-22T21:46:32.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 --force 2024-04-22T21:46:32.477 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 --force 2024-04-22T21:46:32.747 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T21:46:32.748 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T21:46:32.763 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T21:46:32.777 INFO:tasks.cephadm:Teardown complete 2024-04-22T21:46:32.778 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T21:46:32.787 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T21:46:32.788 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T21:46:32.806 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T21:46:32.817 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-04-22T21:46:32.821 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T21:46:32.821 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-04-22T21:46:32.821 INFO:teuthology.orchestra.run.smithi165.stdout:^- hv01.front.sepia.ceph.com 2 6 377 43 +1596us[+1596us] +/- 589ms 2024-04-22T21:46:32.821 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -333us[ -327us] +/- 56ms 2024-04-22T21:46:32.821 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +401us[ +401us] +/- 66ms 2024-04-22T21:46:32.822 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:46:32.830 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-04-22T21:46:32.834 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T21:46:32.835 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-22T21:46:32.835 INFO:teuthology.orchestra.run.smithi188.stdout:^- hv01.front.sepia.ceph.com 2 6 377 41 +1594us[+1594us] +/- 589ms 2024-04-22T21:46:32.835 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -335us[ -329us] +/- 56ms 2024-04-22T21:46:32.835 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +406us[ +411us] +/- 66ms 2024-04-22T21:46:32.835 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T21:46:32.835 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T21:46:32.844 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T21:46:32.845 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T21:46:32.854 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T21:46:32.928 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T21:46:32.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T21:46:33.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:46:33.017 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T21:46:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:46:33.043 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T21:46:33.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T21:46:33.061 INFO:teuthology.task.internal:Duration was 1033.755203 seconds 2024-04-22T21:46:33.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T21:46:33.070 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T21:46:33.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T21:46:33.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T21:46:33.106 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:46:33.119 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:46:33.344 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T21:46:33.345 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-04-22T21:46:33.345 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T21:46:33.405 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-04-22T21:46:33.405 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T21:46:33.426 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T21:46:33.426 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T21:46:33.448 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T21:46:33.490 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T21:46:33.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T21:46:33.512 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T21:46:33.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T21:46:33.808 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T21:46:33.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T21:46:33.832 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T21:46:33.855 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T21:46:33.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T21:46:33.875 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T21:46:33.894 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-04-22T21:46:33.918 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-04-22T21:46:33.952 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T21:46:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:46:33.967 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T21:46:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:46:34.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T21:46:34.017 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T21:46:34.018 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763/remote/smithi165 2024-04-22T21:46:34.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T21:46:34.058 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763/remote/smithi188 2024-04-22T21:46:34.058 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T21:46:34.093 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T21:46:34.093 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T21:46:34.096 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T21:46:34.141 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T21:46:34.152 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T21:46:34.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T21:46:34.161 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T21:46:34.161 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T21:46:34.164 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T21:46:34.176 INFO:teuthology.orchestra.run.smithi165.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 21:46 /home/ubuntu/cephtest 2024-04-22T21:46:34.196 INFO:teuthology.orchestra.run.smithi188.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 21:46 /home/ubuntu/cephtest 2024-04-22T21:46:34.197 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T21:46:34.208 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T21:46:34.228 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T21:46:34.260 DEBUG:teuthology.nuke:shortname: smithi165 2024-04-22T21:46:34.261 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T21:46:34.284 DEBUG:teuthology.nuke:shortname: smithi188 2024-04-22T21:46:34.284 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T21:46:34.288 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:23:14.315909', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO4HzJW48ghDW1vTuINRaToOKaci8aNMe27peObMwgOlmj3xo1tOOP5B6v0WwbEHJfuKk/FxMHMkEC8MMDkNwA4='} 2024-04-22T21:46:34.308 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668763', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:23:14.313914', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEeQqi5NdsyclP/NHZD0RJslPz89iyER8LjpFHca7zg7u1GcO7Auqf0dAsT3PISN4PZNZXvAmDgFsr91GsIac74='} 2024-04-22T21:46:34.339 INFO:teuthology.orchestra.console.smithi165:Power off 2024-04-22T21:46:34.340 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T21:46:34.363 DEBUG:teuthology.orchestra.console.smithi165:power off output: Chassis Power Control: Down/Off 2024-04-22T21:46:34.364 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:46:34.382 INFO:teuthology.orchestra.console.smithi188:Power off 2024-04-22T21:46:34.382 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T21:46:34.392 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-04-22T21:46:34.405 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-04-22T21:46:34.405 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:46:34.428 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-22T21:46:38.393 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:46:38.430 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:46:38.503 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-04-22T21:46:38.540 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-22T21:46:42.506 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:46:42.541 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T21:46:42.617 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is off 2024-04-22T21:46:42.651 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-04-22T21:46:42.718 INFO:teuthology.orchestra.console.smithi165:Power off completed 2024-04-22T21:46:42.752 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-04-22T21:46:42.857 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1033.7552032470703 failure_reason: 'Command failed on smithi165 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dccb09a4-00ef-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \''ceph versions | jq -e \''"\''"\''.overall | keys\''"\''"\'' | grep $sha1\''''' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4d0b013d53194c8b972ba2500031f18c status: fail success: false 2024-04-22T21:46:42.858 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T21:46:42.947 INFO:teuthology.run:FAIL