2024-05-03T10:14:01.750 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-03T10:14:01.751 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290 --owner scheduled_gabrioux@teuthology --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} --name gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi --verbose -- /home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290/orig.config.yaml 2024-05-03T10:14:01.797 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-03T10:14:01.885 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290 branch: wip-guits-testing-2024-05-02-0628 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} email: gabrioux@ibm.com first_in_suite: false job_id: '7687290' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-2024-05-02-0628 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: f8e6d2d721bc96a93915917c59bc3d287d57263d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f8e6d2d721bc96a93915917c59bc3d287d57263d workunit: branch: wip-guits-testing-2024-05-02-0628 sha1: f8e6d2d721bc96a93915917c59bc3d287d57263d owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 2184 sha1: f8e6d2d721bc96a93915917c59bc3d287d57263d sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-2024-05-02-0628 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_f8e6d2d721bc96a93915917c59bc3d287d57263d/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: f8e6d2d721bc96a93915917c59bc3d287d57263d targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLOkh42FXMbxs2iYGdvedXuJFylnSWyEHsKB+Ia9rhtT+hfknqm4PPcd39iLIHdV5Qjxo7slCW2ubNbAo715w7U= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOEH7A8vm25PdYEe27lstXC5VT3fC2iAWSBCF7kT7dyraSFyWG8nz+a3YJ6nftkTcPWQfxlDAaq9QNTT16xYlE= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-03_06:31:58 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-03T10:14:01.885 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_f8e6d2d721bc96a93915917c59bc3d287d57263d/qa; will attempt to use it 2024-05-03T10:14:01.886 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_f8e6d2d721bc96a93915917c59bc3d287d57263d/qa/tasks 2024-05-03T10:14:01.886 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-03T10:14:01.887 INFO:teuthology.task.internal:Checking packages... 2024-05-03T10:14:01.911 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f8e6d2d721bc96a93915917c59bc3d287d57263d' 2024-05-03T10:14:01.911 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-03T10:14:01.911 INFO:teuthology.packaging:ref: None 2024-05-03T10:14:01.911 INFO:teuthology.packaging:tag: None 2024-05-03T10:14:01.911 INFO:teuthology.packaging:branch: wip-guits-testing-2024-05-02-0628 2024-05-03T10:14:01.911 INFO:teuthology.packaging:sha1: f8e6d2d721bc96a93915917c59bc3d287d57263d 2024-05-03T10:14:01.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-2024-05-02-0628 2024-05-03T10:14:02.138 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3353.gf8e6d2d7 2024-05-03T10:14:02.140 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-03T10:14:02.146 INFO:teuthology.task.internal:no buildpackages task found 2024-05-03T10:14:02.146 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-03T10:14:02.159 INFO:teuthology.task.internal:Saving configuration 2024-05-03T10:14:02.176 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-03T10:14:02.182 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T10:14:02.206 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 10:09:19.775635', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLOkh42FXMbxs2iYGdvedXuJFylnSWyEHsKB+Ia9rhtT+hfknqm4PPcd39iLIHdV5Qjxo7slCW2ubNbAo715w7U='} 2024-05-03T10:14:02.230 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 10:09:19.774458', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOEH7A8vm25PdYEe27lstXC5VT3fC2iAWSBCF7kT7dyraSFyWG8nz+a3YJ6nftkTcPWQfxlDAaq9QNTT16xYlE='} 2024-05-03T10:14:02.230 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-03T10:14:02.236 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-05-03T10:14:02.236 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-05-03T10:14:02.236 INFO:teuthology.run_tasks:Running task console_log... 2024-05-03T10:14:02.304 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcdeff2f0d0>, signals=[15]) 2024-05-03T10:14:02.305 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-03T10:14:02.311 INFO:teuthology.task.internal:Opening connections... 2024-05-03T10:14:02.311 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-05-03T10:14:02.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T10:14:02.392 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-05-03T10:14:02.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T10:14:02.469 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-03T10:14:02.476 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-05-03T10:14:02.494 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-05-03T10:14:02.494 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="CentOS Stream" 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="9" 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:ID="centos" 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="rhel fedora" 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="9" 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el9" 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:LOGO="fedora-logo-icon" 2024-05-03T10:14:02.551 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-03T10:14:02.552 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://centos.org/" 2024-05-03T10:14:02.552 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-03T10:14:02.552 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-03T10:14:02.552 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-03T10:14:02.552 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-05-03T10:14:02.581 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-05-03T10:14:02.599 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-05-03T10:14:02.599 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-05-03T10:14:02.656 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-05-03T10:14:02.656 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-05-03T10:14:02.656 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-05-03T10:14:02.656 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-05-03T10:14:02.656 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-05-03T10:14:02.656 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-05-03T10:14:02.657 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-03T10:14:02.657 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-05-03T10:14:02.657 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-05-03T10:14:02.657 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-03T10:14:02.657 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-05-03T10:14:02.657 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-03T10:14:02.657 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-03T10:14:02.657 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-03T10:14:02.657 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-05-03T10:14:02.681 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-03T10:14:02.689 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-03T10:14:02.715 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-03T10:14:02.715 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-05-03T10:14:02.718 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-05-03T10:14:02.734 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-03T10:14:02.739 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-03T10:14:02.740 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-05-03T10:14:02.775 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-05-03T10:14:02.875 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-03T10:14:02.928 INFO:teuthology.run_tasks:Running task kernel... 2024-05-03T10:14:02.942 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-03T10:14:02.942 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-03T10:14:02.943 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-03T10:14:02.943 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-03T10:14:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T10:14:02.960 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-05-03T10:14:02.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T10:14:02.961 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-05-03T10:14:03.017 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-437.el9.x86_64 2024-05-03T10:14:03.017 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-437.el9.x86_64 2024-05-03T10:14:03.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-05-03T10:14:03.019 INFO:teuthology.orchestra.run.smithi043.stdout:5.14.0-437.el9.x86_64 2024-05-03T10:14:03.019 INFO:teuthology.task.kernel:Running kernel on smithi043: 5.14.0-437.el9.x86_64 2024-05-03T10:14:03.019 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum install -y kernel 2024-05-03T10:14:05.638 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-05-03T10:14:06.730 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.0 MB 00:02 2024-05-03T10:14:07.581 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-05-03T10:14:08.747 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-05-03T10:14:11.707 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-03T10:14:12.940 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-05-03T10:14:13.657 INFO:teuthology.orchestra.run.smithi043.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-05-03T10:14:14.473 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 16 kB 00:00 2024-05-03T10:14:15.636 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-05-03T10:14:16.050 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-03T10:14:20.917 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-05-03T10:14:21.275 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-05-03T10:14:22.832 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-03T10:14:22.832 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-03T10:14:22.930 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-05-03T10:14:22.935 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-05-03T10:14:22.935 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-05-03T10:14:23.043 DEBUG:teuthology.orchestra.run.smithi043:> echo no | sudo yum reinstall kernel || true 2024-05-03T10:14:23.198 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-03T10:14:23.198 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-03T10:14:23.291 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-03T10:14:23.292 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-05-03T10:14:23.292 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-03T10:14:23.391 DEBUG:teuthology.orchestra.run.smithi181:> echo no | sudo yum reinstall kernel || true 2024-05-03T10:14:23.567 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:03 ago on Fri 03 May 2024 10:14:20 AM UTC. 2024-05-03T10:14:23.691 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-05-03T10:14:23.705 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-05-03T10:14:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-05-03T10:14:23.705 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-05-03T10:14:23.705 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2024-05-03T10:14:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-03T10:14:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:14:23.706 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-05-03T10:14:23.706 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-05-03T10:14:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:14:23.706 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 255 k 2024-05-03T10:14:23.706 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2024-05-03T10:14:23.706 INFO:teuthology.orchestra.run.smithi043.stderr:Operation aborted. 2024-05-03T10:14:23.764 INFO:teuthology.orchestra.run.smithi043.stdout:Is this ok [y/N]: 2024-05-03T10:14:23.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum reinstall -y kernel || true 2024-05-03T10:14:23.893 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:02 ago on Fri 03 May 2024 10:14:21 AM UTC. 2024-05-03T10:14:24.012 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-03T10:14:24.025 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-03T10:14:24.025 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-05-03T10:14:24.025 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-03T10:14:24.025 INFO:teuthology.orchestra.run.smithi181.stdout:Reinstalling: 2024-05-03T10:14:24.026 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-03T10:14:24.026 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:14:24.026 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-03T10:14:24.026 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-03T10:14:24.026 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:14:24.026 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 255 k 2024-05-03T10:14:24.026 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 0 2024-05-03T10:14:24.026 INFO:teuthology.orchestra.run.smithi181.stderr:Operation aborted. 2024-05-03T10:14:24.081 INFO:teuthology.orchestra.run.smithi181.stdout:Is this ok [y/N]: 2024-05-03T10:14:24.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum reinstall -y kernel || true 2024-05-03T10:14:24.278 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:04 ago on Fri 03 May 2024 10:14:20 AM UTC. 2024-05-03T10:14:24.398 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalling: 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 255 k 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 0 2024-05-03T10:14:24.399 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-05-03T10:14:24.577 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:03 ago on Fri 03 May 2024 10:14:21 AM UTC. 2024-05-03T10:14:24.693 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-05-03T10:14:24.693 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-03T10:14:24.693 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout:Reinstalling: 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 255 k 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 0 2024-05-03T10:14:24.694 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-05-03T10:14:25.069 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-437.el9.x86_64.rpm 2.5 MB/s | 255 kB 00:00 2024-05-03T10:14:25.069 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-05-03T10:14:25.069 INFO:teuthology.orchestra.run.smithi181.stdout:Total 680 kB/s | 255 kB 00:00 2024-05-03T10:14:25.089 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-05-03T10:14:25.265 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-05-03T10:14:25.266 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-05-03T10:14:25.270 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-05-03T10:14:25.271 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-05-03T10:14:25.344 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-437.el9.x86_64.rpm 319 kB/s | 255 kB 00:00 2024-05-03T10:14:25.344 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-05-03T10:14:25.345 INFO:teuthology.orchestra.run.smithi043.stdout:Total 269 kB/s | 255 kB 00:00 2024-05-03T10:14:25.364 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-05-03T10:14:25.409 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-05-03T10:14:25.486 INFO:teuthology.orchestra.run.smithi181.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-03T10:14:25.486 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T10:14:25.564 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-05-03T10:14:25.564 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-05-03T10:14:25.570 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-05-03T10:14:25.570 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-05-03T10:14:25.708 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-05-03T10:14:25.779 INFO:teuthology.orchestra.run.smithi043.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-03T10:14:25.779 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T10:14:26.174 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T10:14:26.174 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-03T10:14:26.518 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T10:14:26.518 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-03T10:14:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T10:14:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:14:26.881 INFO:teuthology.orchestra.run.smithi181.stdout:Reinstalled: 2024-05-03T10:14:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-03T10:14:26.881 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:14:26.881 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-05-03T10:14:27.066 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-05-03T10:14:27.115 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-03T10:14:27.115 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-03T10:14:27.115 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-03T10:14:27.115 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-03T10:14:27.116 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-03T10:14:27.149 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-05-03T10:14:27.168 DEBUG:teuthology.parallel:result is None 2024-05-03T10:14:27.238 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T10:14:27.238 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:14:27.238 INFO:teuthology.orchestra.run.smithi043.stdout:Reinstalled: 2024-05-03T10:14:27.238 INFO:teuthology.orchestra.run.smithi043.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-03T10:14:27.238 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:14:27.238 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-05-03T10:14:27.407 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q kernel | sort -rV | head -n 1 2024-05-03T10:14:27.455 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-03T10:14:27.455 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-03T10:14:27.455 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-03T10:14:27.456 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-03T10:14:27.456 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-03T10:14:27.490 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2024-05-03T10:14:27.519 DEBUG:teuthology.parallel:result is None 2024-05-03T10:14:27.519 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-03T10:14:27.527 INFO:teuthology.task.internal:Creating test directory... 2024-05-03T10:14:27.527 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-03T10:14:27.563 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-03T10:14:27.578 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-03T10:14:27.584 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-03T10:14:27.592 INFO:teuthology.task.internal:Creating archive directory... 2024-05-03T10:14:27.592 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-03T10:14:27.618 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-03T10:14:27.650 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-03T10:14:27.658 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-03T10:14:27.658 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-03T10:14:27.676 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-03T10:14:27.706 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T10:14:27.715 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T10:14:27.716 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T10:14:27.724 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T10:14:27.726 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-03T10:14:27.733 INFO:teuthology.task.internal:Configuring sudo... 2024-05-03T10:14:27.733 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-03T10:14:27.760 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-03T10:14:27.790 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-03T10:14:27.799 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-03T10:14:27.799 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-03T10:14:27.825 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-03T10:14:27.844 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T10:14:27.915 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T10:14:27.967 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T10:14:28.048 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T10:14:28.088 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:14:28.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-03T10:14:28.153 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T10:14:28.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T10:14:28.241 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T10:14:28.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T10:14:28.360 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-03T10:14:28.361 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-03T10:14:28.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-05-03T10:14:28.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-05-03T10:14:28.467 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T10:14:28.499 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T10:14:28.905 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-03T10:14:28.913 INFO:teuthology.task.internal:Starting timer... 2024-05-03T10:14:28.913 INFO:teuthology.run_tasks:Running task pcp... 2024-05-03T10:14:28.927 INFO:teuthology.run_tasks:Running task selinux... 2024-05-03T10:14:28.935 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2024-05-03T10:14:29.001 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2024-05-03T10:14:29.003 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-05-03T10:14:29.060 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-05-03T10:14:29.062 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-03T10:14:29.062 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2024-05-03T10:14:29.085 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2024-05-03T10:14:29.085 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-05-03T10:14:29.110 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-05-03T10:14:29.111 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi043.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-05-03T10:14:29.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-03T10:14:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T10:14:29.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-03T10:14:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T10:14:29.176 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-03T10:14:29.176 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2024-05-03T10:14:29.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-05-03T10:14:29.241 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-03T10:14:29.249 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-03T10:14:29.265 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-03T10:14:29.266 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-03T10:18:24.081 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-05-03T10:18:24.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-05-03T10:18:24.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T10:18:24.167 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-05-03T10:18:24.254 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-05-03T10:18:24.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-05-03T10:18:24.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T10:18:24.331 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-05-03T10:18:24.422 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-05-03T10:18:24.422 INFO:teuthology.run_tasks:Running task clock... 2024-05-03T10:18:24.434 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-03T10:18:24.434 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-03T10:18:24.435 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T10:18:24.437 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-03T10:18:24.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T10:18:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-03T10:18:24.480 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-03T10:18:24.491 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-03T10:18:24.504 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-03T10:18:24.505 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2024-05-03T10:18:24.516 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2024-05-03T10:18:24.527 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-05-03T10:18:24.530 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-03T10:18:24.537 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-05-03T10:18:24.544 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-03T10:18:24.550 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-03T10:18:24.563 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-03T10:18:24.601 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-05-03T10:18:24.602 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T10:18:24.602 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-05-03T10:18:24.603 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:18:24.603 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:18:24.603 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:18:24.603 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:18:24.616 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-05-03T10:18:24.620 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T10:18:24.620 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-05-03T10:18:24.620 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:18:24.620 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:18:24.620 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:18:24.620 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:18:24.621 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-03T10:18:24.726 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'f8e6d2d721bc96a93915917c59bc3d287d57263d'} 2024-05-03T10:18:24.726 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-05-03T10:18:24.727 INFO:tasks.cephadm:Cluster fsid is 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:18:24.727 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-03T10:18:24.727 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.43', 'mon.c': '[v2:172.21.15.43:3301,v1:172.21.15.43:6790]', 'mon.b': '172.21.15.181'} 2024-05-03T10:18:24.727 INFO:tasks.cephadm:First mon is mon.a on smithi043 2024-05-03T10:18:24.728 INFO:tasks.cephadm:First mgr is y 2024-05-03T10:18:24.728 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-03T10:18:24.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo hostname $(hostname -s) 2024-05-03T10:18:24.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2024-05-03T10:18:24.778 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-05-03T10:18:24.779 DEBUG:teuthology.orchestra.run.smithi043:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-03T10:18:25.070 INFO:teuthology.orchestra.run.smithi043.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 3 10:18 /home/ubuntu/cephtest/cephadm 2024-05-03T10:18:25.071 DEBUG:teuthology.orchestra.run.smithi181:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-03T10:18:25.236 INFO:teuthology.orchestra.run.smithi181.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 3 10:18 /home/ubuntu/cephtest/cephadm 2024-05-03T10:18:25.236 DEBUG:teuthology.orchestra.run.smithi043:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-03T10:18:25.253 DEBUG:teuthology.orchestra.run.smithi181:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-03T10:18:25.279 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-05-03T10:18:25.280 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-03T10:18:25.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-03T10:18:26.858 INFO:teuthology.orchestra.run.smithi043.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-03T10:18:26.884 INFO:teuthology.orchestra.run.smithi181.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-03T10:18:45.693 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-05-03T10:18:45.693 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-03T10:18:45.693 INFO:teuthology.orchestra.run.smithi181.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-03T10:18:45.693 INFO:teuthology.orchestra.run.smithi181.stdout: "repo_digests": [ 2024-05-03T10:18:45.694 INFO:teuthology.orchestra.run.smithi181.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-03T10:18:45.694 INFO:teuthology.orchestra.run.smithi181.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-03T10:18:45.694 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-05-03T10:18:45.694 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-05-03T10:18:46.541 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-03T10:18:46.541 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-03T10:18:46.542 INFO:teuthology.orchestra.run.smithi043.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-03T10:18:46.542 INFO:teuthology.orchestra.run.smithi043.stdout: "repo_digests": [ 2024-05-03T10:18:46.542 INFO:teuthology.orchestra.run.smithi043.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-03T10:18:46.542 INFO:teuthology.orchestra.run.smithi043.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-03T10:18:46.542 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-03T10:18:46.543 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-03T10:18:46.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph 2024-05-03T10:18:46.612 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2024-05-03T10:18:46.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /etc/ceph 2024-05-03T10:18:46.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2024-05-03T10:18:46.712 INFO:tasks.cephadm:Writing seed config... 2024-05-03T10:18:46.713 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-05-03T10:18:46.713 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-03T10:18:46.713 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-03T10:18:46.713 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-05-03T10:18:46.714 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-03T10:18:46.714 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-03T10:18:46.714 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-03T10:18:46.714 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-03T10:18:46.714 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-03T10:18:46.714 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:18:46.714 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-03T10:18:46.730 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 79bfd44e-0936-11ef-bc97-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-05-03T10:18:46.731 DEBUG:teuthology.orchestra.run.smithi043:mon.a> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.a.service 2024-05-03T10:18:46.773 DEBUG:teuthology.orchestra.run.smithi043:mgr.y> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mgr.y.service 2024-05-03T10:18:46.816 INFO:tasks.cephadm:Bootstrapping... 2024-05-03T10:18:46.817 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.43 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-03T10:18:47.016 INFO:teuthology.orchestra.run.smithi043.stderr:-------------------------------------------------------------------------------- 2024-05-03T10:18:47.016 INFO:teuthology.orchestra.run.smithi043.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '79bfd44e-0936-11ef-bc97-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.43', '--skip-admin-label'] 2024-05-03T10:18:47.040 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: 4.9.4-dev 2024-05-03T10:18:47.042 INFO:teuthology.orchestra.run.smithi043.stderr:Verifying podman|docker is present... 2024-05-03T10:18:47.065 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: 4.9.4-dev 2024-05-03T10:18:47.066 INFO:teuthology.orchestra.run.smithi043.stderr:Verifying lvm2 is present... 2024-05-03T10:18:47.067 INFO:teuthology.orchestra.run.smithi043.stderr:Verifying time synchronization is in place... 2024-05-03T10:18:47.074 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-03T10:18:47.080 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: inactive 2024-05-03T10:18:47.087 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: enabled 2024-05-03T10:18:47.093 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: active 2024-05-03T10:18:47.093 INFO:teuthology.orchestra.run.smithi043.stderr:Unit chronyd.service is enabled and running 2024-05-03T10:18:47.094 INFO:teuthology.orchestra.run.smithi043.stderr:Repeating the final host check... 2024-05-03T10:18:47.120 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: 4.9.4-dev 2024-05-03T10:18:47.122 INFO:teuthology.orchestra.run.smithi043.stderr:podman (/bin/podman) version 4.9.4 is present 2024-05-03T10:18:47.122 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl is present 2024-05-03T10:18:47.122 INFO:teuthology.orchestra.run.smithi043.stderr:lvcreate is present 2024-05-03T10:18:47.128 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-03T10:18:47.134 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: inactive 2024-05-03T10:18:47.141 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: enabled 2024-05-03T10:18:47.147 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: active 2024-05-03T10:18:47.148 INFO:teuthology.orchestra.run.smithi043.stderr:Unit chronyd.service is enabled and running 2024-05-03T10:18:47.148 INFO:teuthology.orchestra.run.smithi043.stderr:Host looks OK 2024-05-03T10:18:47.148 INFO:teuthology.orchestra.run.smithi043.stderr:Cluster fsid: 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:18:47.148 INFO:teuthology.orchestra.run.smithi043.stderr:Acquiring lock 140701292121968 on /run/cephadm/79bfd44e-0936-11ef-bc97-c7b262605968.lock 2024-05-03T10:18:47.148 INFO:teuthology.orchestra.run.smithi043.stderr:Lock 140701292121968 acquired on /run/cephadm/79bfd44e-0936-11ef-bc97-c7b262605968.lock 2024-05-03T10:18:47.149 INFO:teuthology.orchestra.run.smithi043.stderr:Verifying IP 172.21.15.43 port 3300 ... 2024-05-03T10:18:47.149 INFO:teuthology.orchestra.run.smithi043.stderr:Verifying IP 172.21.15.43 port 6789 ... 2024-05-03T10:18:47.149 INFO:teuthology.orchestra.run.smithi043.stderr:Base mon IP is 172.21.15.43, final addrv is [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-05-03T10:18:47.151 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.43 metric 100 2024-05-03T10:18:47.151 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.43 metric 100 2024-05-03T10:18:47.154 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-05-03T10:18:47.154 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-05-03T10:18:47.154 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-05-03T10:18:47.156 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-03T10:18:47.157 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-05-03T10:18:47.157 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-03T10:18:47.157 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-03T10:18:47.157 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1358/64 scope link noprefixroute 2024-05-03T10:18:47.157 INFO:teuthology.orchestra.run.smithi043.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-03T10:18:47.157 INFO:teuthology.orchestra.run.smithi043.stderr:Mon IP `172.21.15.43` is in CIDR network `172.21.0.0/20` 2024-05-03T10:18:47.158 INFO:teuthology.orchestra.run.smithi043.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-03T10:18:47.158 INFO:teuthology.orchestra.run.smithi043.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-03T10:18:47.264 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-05-03T10:18:47.676 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: Getting image source signatures 2024-05-03T10:18:47.676 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-05-03T10:18:47.676 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-05-03T10:18:47.677 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-05-03T10:18:47.677 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-05-03T10:18:47.677 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-05-03T10:18:47.677 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-03T10:18:47.843 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: Writing manifest to image destination 2024-05-03T10:18:48.404 INFO:teuthology.orchestra.run.smithi043.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-03T10:18:51.865 INFO:teuthology.orchestra.run.smithi043.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-03T10:18:57.722 INFO:teuthology.orchestra.run.smithi043.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-03T10:18:57.723 INFO:teuthology.orchestra.run.smithi043.stderr:Extracting ceph user uid/gid from container image... 2024-05-03T10:18:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:stat: 167 167 2024-05-03T10:19:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:Creating initial keys... 2024-05-03T10:19:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph-authtool: AQAUujRmN56dIhAALkmgfuZouv2xB+cxuU+5PA== 2024-05-03T10:19:01.554 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph-authtool: AQAVujRmjpsPHBAALLNmUQCZDbYLrWlu0NVnaw== 2024-05-03T10:19:02.460 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph-authtool: AQAWujRm+3RuFhAAntKkPx6apE0/kmdZ9WKs+A== 2024-05-03T10:19:02.959 INFO:teuthology.orchestra.run.smithi043.stderr:Creating initial monmap... 2024-05-03T10:19:03.392 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-03T10:19:03.392 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-05-03T10:19:03.392 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:19:03.392 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-03T10:19:03.865 INFO:teuthology.orchestra.run.smithi043.stderr:monmaptool for a [v2:172.21.15.43:3300,v1:172.21.15.43:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-03T10:19:03.865 INFO:teuthology.orchestra.run.smithi043.stderr:setting min_mon_release = octopus 2024-05-03T10:19:03.865 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/monmaptool: set fsid to 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:19:03.865 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-03T10:19:03.866 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:19:03.866 INFO:teuthology.orchestra.run.smithi043.stderr:Creating mon... 2024-05-03T10:19:05.408 INFO:teuthology.orchestra.run.smithi043.stderr:create mon.a on 2024-05-03T10:19:05.616 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-03T10:19:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-79bfd44e-0936-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-79bfd44e-0936-11ef-bc97-c7b262605968.target. 2024-05-03T10:19:05.769 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-79bfd44e-0936-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-79bfd44e-0936-11ef-bc97-c7b262605968.target. 2024-05-03T10:19:06.158 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: Failed to reset failed state of unit ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.a.service: Unit ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.a.service not loaded. 2024-05-03T10:19:06.166 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: Created symlink /etc/systemd/system/ceph-79bfd44e-0936-11ef-bc97-c7b262605968.target.wants/ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.a.service → /etc/systemd/system/ceph-79bfd44e-0936-11ef-bc97-c7b262605968@.service. 2024-05-03T10:19:06.865 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: disabled 2024-05-03T10:19:06.872 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: inactive 2024-05-03T10:19:06.872 INFO:teuthology.orchestra.run.smithi043.stderr:firewalld.service is not enabled 2024-05-03T10:19:06.872 INFO:teuthology.orchestra.run.smithi043.stderr:Not possible to enable service . firewalld.service is not available 2024-05-03T10:19:06.873 INFO:teuthology.orchestra.run.smithi043.stderr:Waiting for mon to start... 2024-05-03T10:19:06.873 INFO:teuthology.orchestra.run.smithi043.stderr:Waiting for mon... 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: cluster: 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: id: 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: services: 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.566612s) 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mgr: no daemons active 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: data: 2024-05-03T10:19:07.394 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-05-03T10:19:07.395 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-05-03T10:19:07.395 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-05-03T10:19:07.395 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: pgs: 2024-05-03T10:19:07.395 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:07.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:07 smithi043 ceph-mon[22054]: from='client.? 172.21.15.43:0/2828693968' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-03T10:19:07.919 INFO:teuthology.orchestra.run.smithi043.stderr:mon is available 2024-05-03T10:19:07.919 INFO:teuthology.orchestra.run.smithi043.stderr:Assimilating anything we can from ceph.conf... 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: [global] 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: fsid = 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.43:3300,v1:172.21.15.43:6789] 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: [mgr] 2024-05-03T10:19:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = True 2024-05-03T10:19:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-03T10:19:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: [osd] 2024-05-03T10:19:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-03T10:19:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-03T10:19:09.000 INFO:teuthology.orchestra.run.smithi043.stderr:Generating new minimal ceph.conf... 2024-05-03T10:19:09.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:08 smithi043 ceph-mon[22054]: from='client.? 172.21.15.43:0/2551126698' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-03T10:19:09.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:08 smithi043 ceph-mon[22054]: from='client.? 172.21.15.43:0/2551126698' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-03T10:19:10.040 INFO:teuthology.orchestra.run.smithi043.stderr:Restarting the monitor... 2024-05-03T10:19:10.058 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:09 smithi043 ceph-mon[22054]: from='client.? 172.21.15.43:0/3356856977' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:19:10.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:10 smithi043 systemd[1]: Stopping Ceph mon.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:19:10.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:10 smithi043 bash[22273]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon.a" found: no such container 2024-05-03T10:19:10.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:10 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a[22048]: 2024-05-03T10:19:10.189+0000 7f2de0483700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:19:10.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:10 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a[22048]: 2024-05-03T10:19:10.189+0000 7f2de0483700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-03T10:19:10.346 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:10 smithi043 podman[22285]: 2024-05-03 10:19:10.344088968 +0000 UTC m=+0.250548216 container died a71f1dc1b0b56d896b444924a8d532cafcef477e0b668efede32264689a6ecf5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a, release=754, version=8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, maintainer=Guillaume Abrioux , architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_CLEAN=True, ceph=True, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-03T10:19:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:10 smithi043 podman[22285]: 2024-05-03 10:19:10.822733713 +0000 UTC m=+0.729192968 container cleanup a71f1dc1b0b56d896b444924a8d532cafcef477e0b668efede32264689a6ecf5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a, release=754, vcs-type=git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, distribution-scope=public, vendor=Red Hat, Inc., architecture=x86_64, name=centos-stream, version=8, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8) 2024-05-03T10:19:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:10 smithi043 bash[22285]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a 2024-05-03T10:19:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:10 smithi043 bash[22311]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon.a" found: no such container 2024-05-03T10:19:11.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 podman[22299]: 2024-05-03 10:19:11.057131093 +0000 UTC m=+0.708334528 container remove a71f1dc1b0b56d896b444924a8d532cafcef477e0b668efede32264689a6ecf5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, vcs-type=git, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, release=754, ceph=True, GIT_CLEAN=True, io.openshift.expose-services=, RELEASE=HEAD, io.buildah.version=1.19.8, name=centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-03T10:19:11.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-03T10:19:11.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 systemd[1]: Stopped Ceph mon.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:19:11.318 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 systemd[1]: Starting Ceph mon.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:19:11.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 podman[22385]: 2024-05-03 10:19:11.316698781 +0000 UTC m=+0.113870047 container create 92837a4d1736fd5e9f8921c7aa0ad49e5a4fa9213ea28f70fadf0b8f67b52791 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a, ceph=True, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, version=8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, release=754, io.buildah.version=1.19.8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-03T10:19:11.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 podman[22385]: 2024-05-03 10:19:11.222777755 +0000 UTC m=+0.019949055 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-05-03T10:19:11.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 podman[22385]: 2024-05-03 10:19:11.51056134 +0000 UTC m=+0.307732609 container init 92837a4d1736fd5e9f8921c7aa0ad49e5a4fa9213ea28f70fadf0b8f67b52791 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, release=754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, ceph=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, name=centos-stream, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8) 2024-05-03T10:19:11.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 podman[22385]: 2024-05-03 10:19:11.513539499 +0000 UTC m=+0.310710768 container start 92837a4d1736fd5e9f8921c7aa0ad49e5a4fa9213ea28f70fadf0b8f67b52791 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_CLEAN=True, vcs-type=git, io.openshift.expose-services=, distribution-scope=public, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, architecture=x86_64) 2024-05-03T10:19:11.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: set uid:gid to 167:167 (ceph:ceph) 2024-05-03T10:19:11.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-03T10:19:11.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: pidfile_write: ignore empty --pid-file 2024-05-03T10:19:11.587 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: load: jerasure load: lrc 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: RocksDB version: 6.15.5 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Compile date Apr 18 2022 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: DB SUMMARY 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: DB Session ID: NJVDEHEZI7EMGRQA8YG1 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: CURRENT file: CURRENT 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: IDENTITY file: IDENTITY 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82635 ; 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.error_if_exists: 0 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.create_if_missing: 0 2024-05-03T10:19:11.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.paranoid_checks: 1 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.env: 0x564ebfcc6860 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.fs: Posix File System 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.info_log: 0x564ec1cdddc0 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_file_opening_threads: 16 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.statistics: (nil) 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.use_fsync: 0 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_log_file_size: 0 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.keep_log_file_num: 1000 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.recycle_log_file_num: 0 2024-05-03T10:19:11.589 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.allow_fallocate: 1 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.allow_mmap_reads: 0 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.allow_mmap_writes: 0 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.use_direct_reads: 0 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.create_missing_column_families: 0 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.db_log_dir: 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-03T10:19:11.590 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.advise_random_on_open: 1 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.db_write_buffer_size: 0 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.write_buffer_manager: 0x564ec1dce2a0 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.rate_limiter: (nil) 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.wal_recovery_mode: 2 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.enable_thread_tracking: 0 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.enable_pipelined_write: 0 2024-05-03T10:19:11.591 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.unordered_write: 0 2024-05-03T10:19:11.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-03T10:19:11.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-03T10:19:11.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-03T10:19:11.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-03T10:19:11.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.row_cache: None 2024-05-03T10:19:11.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.wal_filter: None 2024-05-03T10:19:11.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-03T10:19:11.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.allow_ingest_behind: 0 2024-05-03T10:19:11.592 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.preserve_deletes: 0 2024-05-03T10:19:11.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.two_write_queues: 0 2024-05-03T10:19:11.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.manual_wal_flush: 0 2024-05-03T10:19:11.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.atomic_flush: 0 2024-05-03T10:19:11.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-03T10:19:11.593 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.log_readahead_size: 0 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.best_efforts_recovery: 0 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.allow_data_in_errors: 0 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.db_host_id: __hostname__ 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_background_jobs: 2 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_background_compactions: -1 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_subcompactions: 1 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-03T10:19:11.594 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_total_wal_size: 0 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_open_files: -1 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bytes_per_sync: 0 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_readahead_size: 0 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_background_flushes: -1 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Compression algorithms supported: 2024-05-03T10:19:11.595 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: kZSTD supported: 0 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: kXpressCompression supported: 0 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: kLZ4HCCompression supported: 1 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: kLZ4Compression supported: 1 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: kBZip2Compression supported: 0 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: kZlibCompression supported: 1 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: kSnappyCompression supported: 1 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-05-03T10:19:11.596 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.merge_operator: 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_filter: None 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_filter_factory: None 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.sst_partitioner_factory: None 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564ec1ca9d00) 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks: 1 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout: pin_top_level_index_and_filter: 1 2024-05-03T10:19:11.597 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_type: 0 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_index_type: 0 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_shortening: 1 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: hash_index_allow_collision: 1 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: checksum: 1 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: no_block_cache: 0 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache: 0x564ec1d14170 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_name: BinnedLRUCache 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_options: 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: capacity : 536870912 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: num_shard_bits : 4 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: strict_capacity_limit : 0 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: high_pri_pool_ratio: 0.000 2024-05-03T10:19:11.598 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_cache_compressed: (nil) 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: persistent_cache: (nil) 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size: 4096 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_size_deviation: 10 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_restart_interval: 16 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: index_block_restart_interval: 1 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: metadata_block_size: 4096 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: partition_filters: 0 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: use_delta_encoding: 1 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: whole_key_filtering: 1 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: verify_compression: 0 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: read_amp_bytes_per_bit: 0 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: format_version: 4 2024-05-03T10:19:11.599 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_index_compression: 1 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout: block_align: 0 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.write_buffer_size: 33554432 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_write_buffer_number: 2 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compression: NoCompression 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bottommost_compression: Disabled 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.prefix_extractor: nullptr 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.num_levels: 7 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-03T10:19:11.600 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-03T10:19:11.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-03T10:19:11.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-03T10:19:11.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-03T10:19:11.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-03T10:19:11.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-03T10:19:11.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-03T10:19:11.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-03T10:19:11.601 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compression_opts.level: 32767 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compression_opts.strategy: 0 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compression_opts.enabled: false 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.target_file_size_base: 67108864 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-03T10:19:11.602 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.arena_block_size: 4194304 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.disable_auto_compactions: 0 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-03T10:19:11.603 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.table_properties_collectors: 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.inplace_update_support: 0 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-03T10:19:11.604 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.bloom_locality: 0 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.max_successive_merges: 0 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.paranoid_file_checks: 0 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.force_consistency_checks: 1 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.report_bg_io_stats: 0 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.ttl: 2592000 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.enable_blob_files: false 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.min_blob_size: 0 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.blob_file_size: 268435456 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-03T10:19:11.605 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714731551539645, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714731551540443, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79185, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77448, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71757, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1714731551, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "44248424-7cf3-4971-a014-b4f38320d794", "db_session_id": "NJVDEHEZI7EMGRQA8YG1"}} 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714731551540796, "job": 1, "event": "recovery_finished"} 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x564ec1cfaa80 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: DB pointer 0x564ec1d0a000 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** DB Stats ** 2024-05-03T10:19:11.606 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-03T10:19:11.607 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.8 0.00 0.00 1 0.001 0 0 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative compaction: 0.00 GB write, 24.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval compaction: 0.00 GB write, 24.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-03T10:19:11.608 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 138.8 0.00 0.00 1 0.001 0 0 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 138.8 0.00 0.00 1 0.001 0 0 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-03T10:19:11.609 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout: Cumulative compaction: 0.00 GB write, 24.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: starting mon.a rank 0 at public addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] at bind addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mon.a@-1(???) e1 preinit fsid 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mon.a@-1(???).mds e1 new map 2024-05-03T10:19:11.610 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mon.a@-1(???).mds e1 print_map 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout: e1 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout: legacy client fscid: -1 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout: 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout: No filesystems configured 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 bash[22385]: 92837a4d1736fd5e9f8921c7aa0ad49e5a4fa9213ea28f70fadf0b8f67b52791 2024-05-03T10:19:11.611 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 systemd[1]: Started Ceph mon.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:19:11.612 INFO:teuthology.orchestra.run.smithi043.stderr:Setting mon public_network to 172.21.0.0/20 2024-05-03T10:19:11.906 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-03T10:19:11.906 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: monmap e1: 1 mons at {a=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0]} 2024-05-03T10:19:11.906 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: fsmap 2024-05-03T10:19:11.906 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: osdmap e1: 0 total, 0 up, 0 in 2024-05-03T10:19:11.906 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:11 smithi043 ceph-mon[22404]: mgrmap e1: no daemons active 2024-05-03T10:19:12.597 INFO:teuthology.orchestra.run.smithi043.stderr:Wrote config to /etc/ceph/ceph.conf 2024-05-03T10:19:12.597 INFO:teuthology.orchestra.run.smithi043.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-03T10:19:12.597 INFO:teuthology.orchestra.run.smithi043.stderr:Creating mgr... 2024-05-03T10:19:12.597 INFO:teuthology.orchestra.run.smithi043.stderr:Verifying port 9283 ... 2024-05-03T10:19:12.786 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: Failed to reset failed state of unit ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mgr.y.service: Unit ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mgr.y.service not loaded. 2024-05-03T10:19:12.794 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: Created symlink /etc/systemd/system/ceph-79bfd44e-0936-11ef-bc97-c7b262605968.target.wants/ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-79bfd44e-0936-11ef-bc97-c7b262605968@.service. 2024-05-03T10:19:12.964 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:12 smithi043 systemd[1]: Starting Ceph mgr.y for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:19:13.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:13 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/293611349' entity='client.admin' 2024-05-03T10:19:13.276 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:13 smithi043 podman[22619]: 2024-05-03 10:19:13.175936412 +0000 UTC m=+0.109541629 container create e925f3a624c592bc78db21ce091b563e56d1cbeb538847a4d26d9a397693ad7f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, release=754, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, vendor=Red Hat, Inc., ceph=True, version=8, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-03T10:19:13.276 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:13 smithi043 podman[22619]: 2024-05-03 10:19:13.085133295 +0000 UTC m=+0.018738523 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-05-03T10:19:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: disabled 2024-05-03T10:19:13.508 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: inactive 2024-05-03T10:19:13.509 INFO:teuthology.orchestra.run.smithi043.stderr:firewalld.service is not enabled 2024-05-03T10:19:13.509 INFO:teuthology.orchestra.run.smithi043.stderr:Not possible to enable service . firewalld.service is not available 2024-05-03T10:19:13.518 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: disabled 2024-05-03T10:19:13.526 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: inactive 2024-05-03T10:19:13.527 INFO:teuthology.orchestra.run.smithi043.stderr:firewalld.service is not enabled 2024-05-03T10:19:13.527 INFO:teuthology.orchestra.run.smithi043.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-05-03T10:19:13.527 INFO:teuthology.orchestra.run.smithi043.stderr:Waiting for mgr to start... 2024-05-03T10:19:13.527 INFO:teuthology.orchestra.run.smithi043.stderr:Waiting for mgr... 2024-05-03T10:19:13.653 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:13 smithi043 podman[22619]: 2024-05-03 10:19:13.386800883 +0000 UTC m=+0.320406098 container init e925f3a624c592bc78db21ce091b563e56d1cbeb538847a4d26d9a397693ad7f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y, name=centos-stream, io.buildah.version=1.19.8, ceph=True, GIT_BRANCH=HEAD, release=754, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vcs-type=git, GIT_CLEAN=True, distribution-scope=public, architecture=x86_64, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-03T10:19:13.653 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:13 smithi043 podman[22619]: 2024-05-03 10:19:13.38959662 +0000 UTC m=+0.323201836 container start e925f3a624c592bc78db21ce091b563e56d1cbeb538847a4d26d9a397693ad7f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_BRANCH=HEAD, name=centos-stream, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-05-03T10:19:13.653 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:13 smithi043 bash[22619]: e925f3a624c592bc78db21ce091b563e56d1cbeb538847a4d26d9a397693ad7f 2024-05-03T10:19:13.653 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:13 smithi043 systemd[1]: Started Ceph mgr.y for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:19:13.654 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:13.540+0000 7f2e54515000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T10:19:13.929 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:13.677+0000 7f2e54515000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:19:13.929 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:13.779+0000 7f2e54515000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T10:19:14.199 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:14 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:14.110+0000 7f2e54515000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T10:19:14.199 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:14 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:14.197+0000 7f2e54515000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T10:19:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: { 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "fsid": "79bfd44e-0936-11ef-bc97-c7b262605968", 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "health": { 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "checks": {}, 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mutes": [] 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum": [ 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 0 2024-05-03T10:19:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "a" 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "monmap": { 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osdmap": { 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-03T10:19:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-03T10:19:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-03T10:19:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-03T10:19:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "pgmap": { 2024-05-03T10:19:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-03T10:19:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "fsmap": { 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "available": false, 2024-05-03T10:19:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "modules": [ 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "iostat", 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "nfs", 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "restful" 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "services": {} 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "servicemap": { 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "modified": "2024-05-03T10:19:06.824710+0000", 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "services": {} 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-03T10:19:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: } 2024-05-03T10:19:14.478 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:14 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/93029918' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T10:19:14.798 INFO:teuthology.orchestra.run.smithi043.stderr:mgr not available, waiting (1/15)... 2024-05-03T10:19:15.671 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:15 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:15.415+0000 7f2e54515000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T10:19:15.672 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:15 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:15.585+0000 7f2e54515000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T10:19:15.672 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:15 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:15.669+0000 7f2e54515000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T10:19:16.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:15 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:15.994+0000 7f2e54515000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T10:19:16.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:16 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:16.087+0000 7f2e54515000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T10:19:16.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:16 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:16.182+0000 7f2e54515000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T10:19:16.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:16 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:16.431+0000 7f2e54515000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:19:17.150 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:16 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:16.895+0000 7f2e54515000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T10:19:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: { 2024-05-03T10:19:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "fsid": "79bfd44e-0936-11ef-bc97-c7b262605968", 2024-05-03T10:19:17.332 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "health": { 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "checks": {}, 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mutes": [] 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum": [ 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 0 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "a" 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "monmap": { 2024-05-03T10:19:17.333 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osdmap": { 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:17.334 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "pgmap": { 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "fsmap": { 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-03T10:19:17.335 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "available": false, 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "modules": [ 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "iostat", 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "nfs", 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "restful" 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "services": {} 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "servicemap": { 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "modified": "2024-05-03T10:19:06.824710+0000", 2024-05-03T10:19:17.336 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "services": {} 2024-05-03T10:19:17.337 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:17.337 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-03T10:19:17.337 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: } 2024-05-03T10:19:17.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:17 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3205450758' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T10:19:17.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:17 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:17.411+0000 7f2e54515000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T10:19:17.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:17 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:17.507+0000 7f2e54515000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T10:19:17.944 INFO:teuthology.orchestra.run.smithi043.stderr:mgr not available, waiting (2/15)... 2024-05-03T10:19:18.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:17 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:17.815+0000 7f2e54515000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T10:19:18.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:17 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:17.907+0000 7f2e54515000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T10:19:18.685 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:18 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:18.331+0000 7f2e54515000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T10:19:18.685 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:18 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:18.468+0000 7f2e54515000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T10:19:18.685 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:18 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:18.564+0000 7f2e54515000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T10:19:18.685 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:18 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:18.682+0000 7f2e54515000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T10:19:19.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:18 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:18.773+0000 7f2e54515000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T10:19:19.749 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:19 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:19.344+0000 7f2e54515000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T10:19:20.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:19 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:19.747+0000 7f2e54515000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T10:19:20.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: Activating manager daemon y 2024-05-03T10:19:20.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: mgrmap e2: y(active, starting, since 0.00294251s) 2024-05-03T10:19:20.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T10:19:20.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T10:19:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T10:19:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:19:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T10:19:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: Manager daemon y is now available 2024-05-03T10:19:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:19:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:19:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' 2024-05-03T10:19:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' 2024-05-03T10:19:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:19 smithi043 ceph-mon[22404]: from='mgr.14100 172.21.15.43:0/2619809454' entity='mgr.y' 2024-05-03T10:19:20.447 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:20.447 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: { 2024-05-03T10:19:20.447 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "fsid": "79bfd44e-0936-11ef-bc97-c7b262605968", 2024-05-03T10:19:20.447 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "health": { 2024-05-03T10:19:20.447 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-03T10:19:20.447 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "checks": {}, 2024-05-03T10:19:20.447 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mutes": [] 2024-05-03T10:19:20.447 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:20.447 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum": [ 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 0 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "a" 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "monmap": { 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osdmap": { 2024-05-03T10:19:20.448 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "pgmap": { 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-03T10:19:20.449 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "fsmap": { 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "available": false, 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "modules": [ 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "iostat", 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "nfs", 2024-05-03T10:19:20.450 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "restful" 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "services": {} 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "servicemap": { 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "modified": "2024-05-03T10:19:06.824710+0000", 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "services": {} 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-03T10:19:20.451 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: } 2024-05-03T10:19:21.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:20 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1637775169' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T10:19:21.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:20 smithi043 ceph-mon[22404]: mgrmap e3: y(active, since 1.00621s) 2024-05-03T10:19:21.024 INFO:teuthology.orchestra.run.smithi043.stderr:mgr not available, waiting (3/15)... 2024-05-03T10:19:23.668 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:23 smithi043 ceph-mon[22404]: mgrmap e4: y(active, since 2s) 2024-05-03T10:19:23.719 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:23.719 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: { 2024-05-03T10:19:23.719 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "fsid": "79bfd44e-0936-11ef-bc97-c7b262605968", 2024-05-03T10:19:23.719 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "health": { 2024-05-03T10:19:23.719 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-03T10:19:23.719 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "checks": {}, 2024-05-03T10:19:23.719 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mutes": [] 2024-05-03T10:19:23.719 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:23.719 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum": [ 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 0 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "a" 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "monmap": { 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osdmap": { 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:23.720 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "pgmap": { 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-03T10:19:23.721 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "fsmap": { 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "available": true, 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "modules": [ 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "iostat", 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "nfs", 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "restful" 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ], 2024-05-03T10:19:23.722 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "services": {} 2024-05-03T10:19:23.723 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:23.723 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "servicemap": { 2024-05-03T10:19:23.723 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-03T10:19:23.723 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "modified": "2024-05-03T10:19:06.824710+0000", 2024-05-03T10:19:23.723 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "services": {} 2024-05-03T10:19:23.723 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: }, 2024-05-03T10:19:23.723 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-03T10:19:23.723 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: } 2024-05-03T10:19:24.373 INFO:teuthology.orchestra.run.smithi043.stderr:mgr is available 2024-05-03T10:19:24.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:24 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3332479434' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T10:19:24.959 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:24.959 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: [global] 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: fsid = 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: [mgr] 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: [osd] 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-03T10:19:24.960 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-03T10:19:25.538 INFO:teuthology.orchestra.run.smithi043.stderr:Enabling cephadm module... 2024-05-03T10:19:25.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:25 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3872084750' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-03T10:19:25.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:25 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3872084750' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-03T10:19:26.832 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:26 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2474924364' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-03T10:19:26.833 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:26 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ignoring --setuser ceph since I am not root 2024-05-03T10:19:26.833 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:26 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ignoring --setgroup ceph since I am not root 2024-05-03T10:19:26.833 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:26 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:26.714+0000 7fd511e9a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T10:19:27.124 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:26 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:26.848+0000 7fd511e9a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:19:27.125 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:26 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:26.949+0000 7fd511e9a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T10:19:27.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:27 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:27.270+0000 7fd511e9a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T10:19:27.392 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:27 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:27.356+0000 7fd511e9a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T10:19:27.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:27 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2474924364' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-03T10:19:27.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:27 smithi043 ceph-mon[22404]: mgrmap e5: y(active, since 6s) 2024-05-03T10:19:27.863 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: { 2024-05-03T10:19:27.863 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 5, 2024-05-03T10:19:27.863 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "available": true, 2024-05-03T10:19:27.863 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-03T10:19:27.864 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-03T10:19:27.864 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: } 2024-05-03T10:19:28.418 INFO:teuthology.orchestra.run.smithi043.stderr:Waiting for the mgr to restart... 2024-05-03T10:19:28.418 INFO:teuthology.orchestra.run.smithi043.stderr:Waiting for mgr epoch 5... 2024-05-03T10:19:28.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:28 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3634921495' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-03T10:19:28.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:28 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:28.569+0000 7fd511e9a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T10:19:28.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:28 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:28.739+0000 7fd511e9a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T10:19:29.151 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:28 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:28.825+0000 7fd511e9a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T10:19:29.151 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:29 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:29.149+0000 7fd511e9a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T10:19:29.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:29 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:29.241+0000 7fd511e9a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T10:19:29.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:29 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:29.337+0000 7fd511e9a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T10:19:30.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:29 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:29.585+0000 7fd511e9a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:19:30.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:30 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:30.041+0000 7fd511e9a000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T10:19:30.942 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:30 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:30.541+0000 7fd511e9a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T10:19:30.942 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:30 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:30.636+0000 7fd511e9a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T10:19:30.942 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:30 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:30.940+0000 7fd511e9a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T10:19:31.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:31 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:31.030+0000 7fd511e9a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T10:19:31.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:31 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:31.444+0000 7fd511e9a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T10:19:31.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:31 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:31.580+0000 7fd511e9a000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T10:19:31.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:31 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:31.675+0000 7fd511e9a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T10:19:32.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:31 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:31.791+0000 7fd511e9a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T10:19:32.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:31 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:31.883+0000 7fd511e9a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T10:19:32.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:32 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:32.446+0000 7fd511e9a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T10:19:33.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:32 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:32.816+0000 7fd511e9a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T10:19:33.881 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: { 2024-05-03T10:19:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-05-03T10:19:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "initialized": true 2024-05-03T10:19:33.882 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: } 2024-05-03T10:19:34.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: Active manager daemon y restarted 2024-05-03T10:19:34.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: Activating manager daemon y 2024-05-03T10:19:34.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: osdmap e2: 0 total, 0 up, 0 in 2024-05-03T10:19:34.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: mgrmap e6: y(active, starting, since 0.0524911s) 2024-05-03T10:19:34.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:19:34.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: Manager daemon y is now available 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:19:34.246 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:33 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:19:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:mgr epoch 5 is available 2024-05-03T10:19:34.490 INFO:teuthology.orchestra.run.smithi043.stderr:Setting orchestrator backend to cephadm... 2024-05-03T10:19:35.159 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:34 smithi043 ceph-mon[22404]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-03T10:19:35.159 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:34 smithi043 ceph-mon[22404]: mgrmap e7: y(active, since 1.0551s) 2024-05-03T10:19:35.159 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:34 smithi043 ceph-mon[22404]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-03T10:19:35.159 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:34 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:19:34] ENGINE Bus STARTING 2024-05-03T10:19:35.159 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:35 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:19:35] ENGINE Serving on https://172.21.15.43:7150 2024-05-03T10:19:35.159 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:35 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:19:35] ENGINE Bus STARTED 2024-05-03T10:19:35.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:35 smithi043 ceph-mon[22404]: mgrmap e8: y(active, since 2s) 2024-05-03T10:19:35.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:35 smithi043 ceph-mon[22404]: [03/May/2024:10:19:34] ENGINE Bus STARTING 2024-05-03T10:19:35.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:35 smithi043 ceph-mon[22404]: [03/May/2024:10:19:35] ENGINE Serving on https://172.21.15.43:7150 2024-05-03T10:19:35.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:35 smithi043 ceph-mon[22404]: [03/May/2024:10:19:35] ENGINE Bus STARTED 2024-05-03T10:19:35.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:35 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:35.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:35 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:19:35.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:35 smithi043 ceph-mon[22404]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:19:35.969 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:35 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:35.970 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:35 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:19:36.383 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: value unchanged 2024-05-03T10:19:36.961 INFO:teuthology.orchestra.run.smithi043.stderr:Generating ssh key... 2024-05-03T10:19:38.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:37 smithi043 ceph-mon[22404]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: Generating public/private rsa key pair. 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: Your identification has been saved in /tmp/tmp8e9yfsl5/key. 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: Your public key has been saved in /tmp/tmp8e9yfsl5/key.pub. 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: The key fingerprint is: 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: SHA256:11uedkYAi7DrHsM9P6n41UIDomY+e+lc1QEhHvDhGbQ ceph-79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: The key's randomart image is: 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: +---[RSA 3072]----+ 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: | oo* +. | 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: | * O + | 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: | o E . o | 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: | . o o . o | 2024-05-03T10:19:38.340 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: | + S . = o .| 2024-05-03T10:19:38.341 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: | + o o o * o | 2024-05-03T10:19:38.341 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: | o =.+ +.= o| 2024-05-03T10:19:38.341 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: | =o= +oo o | 2024-05-03T10:19:38.341 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: | .o=.o... | 2024-05-03T10:19:38.341 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: +----[SHA256]-----+ 2024-05-03T10:19:38.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:38 smithi043 ceph-mon[22404]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:19:38.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:38 smithi043 ceph-mon[22404]: Generating ssh key... 2024-05-03T10:19:38.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:38 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:38.701 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:38 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:39.230 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsUfoskjRV0rkRbU46s/iepzEE36IPq9O8f6CR+kbJl8aWcCjb1EaZhMwoT1VlII5YuxSZr23tjAc+JocXBFpQS5iMdDHBjJfv7+ctbYhjhrbT9bucq4RGbsV/Da9JiYVvxMSwwIQp4sCB5ua/KgWmK3zMGVy8/cPpVi2gvDWXUEMfsCneTgrs/rBw/ZemfbfKjk23ZgHtEtD5uxIXpWidOjTz9e/nIuc1Y+XlsWdizuaTIYvHprEYmIXzXkL4dr9zKKdCno77w6orid2kOSN6xwg8P9RklXwXqBmbbdwof1DlXoAZXXdM7S6xADhYbDjG5yZg8ZnfdJbDUFzRlb4OkzkmPNct0dDJSOIJ4ohhYvpwsxDxfdSWEEakYIFiHJCrjgiy6R0ACNZtjQyb0PbEgTey8TMWQfumzUgl+tg9rT2q3ipoFckQLbz4Euoe33sKdmzXAYRnSs/FY/5EAH+nKQtw37oXNmOk5C5mx+KyWngIL1Pd7gLjGnS/r5xxqFU= ceph-79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:19:39.770 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:39 smithi043 ceph-mon[22404]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:19:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-03T10:19:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:Adding key to root@localhost authorized_keys... 2024-05-03T10:19:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:Adding host smithi043... 2024-05-03T10:19:41.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:40 smithi043 ceph-mon[22404]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi043", "addr": "172.21.15.43", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:19:41.256 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: Added host 'smithi043' with addr '172.21.15.43' 2024-05-03T10:19:41.847 INFO:teuthology.orchestra.run.smithi043.stderr:Deploying unmanaged mon service... 2024-05-03T10:19:42.443 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:42 smithi043 ceph-mon[22404]: Deploying cephadm binary to smithi043 2024-05-03T10:19:42.444 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:42 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:42.444 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:42 smithi043 ceph-mon[22404]: Added host smithi043 2024-05-03T10:19:42.444 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:42 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:19:42.444 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: Scheduled mon update... 2024-05-03T10:19:43.265 INFO:teuthology.orchestra.run.smithi043.stderr:Deploying unmanaged mgr service... 2024-05-03T10:19:43.536 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:43 smithi043 ceph-mon[22404]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:19:43.536 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:43 smithi043 ceph-mon[22404]: Saving service mon spec with placement count:5 2024-05-03T10:19:43.536 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:43 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:43.976 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-05-03T10:19:45.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:44 smithi043 ceph-mon[22404]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:19:45.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:44 smithi043 ceph-mon[22404]: Saving service mgr spec with placement count:2 2024-05-03T10:19:45.128 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:44 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:46.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:46 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3038541541' entity='client.admin' 2024-05-03T10:19:47.338 INFO:teuthology.orchestra.run.smithi043.stderr:Enabling the dashboard module... 2024-05-03T10:19:47.655 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:47 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1084003689' entity='client.admin' 2024-05-03T10:19:47.656 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:47 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:48.968 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:48 smithi043 ceph-mon[22404]: from='mgr.14120 172.21.15.43:0/2226897641' entity='mgr.y' 2024-05-03T10:19:48.968 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:48 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2684486057' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-03T10:19:48.968 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:48 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ignoring --setuser ceph since I am not root 2024-05-03T10:19:48.968 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:48 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ignoring --setgroup ceph since I am not root 2024-05-03T10:19:48.968 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:48 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:48.832+0000 7f7307ab2000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T10:19:49.288 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:48 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:48.965+0000 7f7307ab2000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:19:49.288 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:49 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:49.067+0000 7f7307ab2000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T10:19:49.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:49 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:49.396+0000 7f7307ab2000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T10:19:49.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:49 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:49.483+0000 7f7307ab2000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T10:19:50.040 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:49 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2684486057' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-03T10:19:50.040 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:49 smithi043 ceph-mon[22404]: mgrmap e9: y(active, since 15s) 2024-05-03T10:19:50.041 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: { 2024-05-03T10:19:50.041 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "epoch": 9, 2024-05-03T10:19:50.041 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "available": true, 2024-05-03T10:19:50.041 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-03T10:19:50.041 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-03T10:19:50.041 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: } 2024-05-03T10:19:50.641 INFO:teuthology.orchestra.run.smithi043.stderr:Waiting for the mgr to restart... 2024-05-03T10:19:50.641 INFO:teuthology.orchestra.run.smithi043.stderr:Waiting for mgr epoch 9... 2024-05-03T10:19:50.889 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:50 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2158291409' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-03T10:19:50.889 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:50 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:50.685+0000 7f7307ab2000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T10:19:50.889 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:50 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:50.858+0000 7f7307ab2000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T10:19:51.278 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:50 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:50.948+0000 7f7307ab2000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T10:19:51.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:51 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:51.276+0000 7f7307ab2000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T10:19:51.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:51 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:51.370+0000 7f7307ab2000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T10:19:51.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:51 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:51.467+0000 7f7307ab2000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T10:19:52.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:51 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:51.717+0000 7f7307ab2000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:19:52.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:52 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:52.177+0000 7f7307ab2000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T10:19:53.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:52 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:52.681+0000 7f7307ab2000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T10:19:53.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:52 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:52.777+0000 7f7307ab2000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T10:19:53.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:53.084+0000 7f7307ab2000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T10:19:53.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:53.177+0000 7f7307ab2000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T10:19:53.949 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:53.594+0000 7f7307ab2000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T10:19:53.949 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:53.732+0000 7f7307ab2000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T10:19:53.949 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:53.828+0000 7f7307ab2000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T10:19:53.949 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:53.947+0000 7f7307ab2000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T10:19:54.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:54 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:54.039+0000 7f7307ab2000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T10:19:54.983 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:54 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:54.608+0000 7f7307ab2000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T10:19:55.241 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:54 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:19:54.981+0000 7f7307ab2000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T10:19:55.498 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:19:55] ENGINE Bus STARTING 2024-05-03T10:19:55.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:19:55] ENGINE Serving on https://172.21.15.43:7150 2024-05-03T10:19:55.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:19:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:19:55] ENGINE Bus STARTED 2024-05-03T10:19:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: { 2024-05-03T10:19:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-05-03T10:19:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: "initialized": true 2024-05-03T10:19:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: } 2024-05-03T10:19:56.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: Active manager daemon y restarted 2024-05-03T10:19:56.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: Activating manager daemon y 2024-05-03T10:19:56.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: osdmap e3: 0 total, 0 up, 0 in 2024-05-03T10:19:56.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: mgrmap e10: y(active, starting, since 0.0528998s) 2024-05-03T10:19:56.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:19:56.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T10:19:56.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T10:19:56.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T10:19:56.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: Manager daemon y is now available 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: [03/May/2024:10:19:55] ENGINE Bus STARTING 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: [03/May/2024:10:19:55] ENGINE Serving on https://172.21.15.43:7150 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: [03/May/2024:10:19:55] ENGINE Bus STARTED 2024-05-03T10:19:56.163 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:19:56.690 INFO:teuthology.orchestra.run.smithi043.stderr:mgr epoch 9 is available 2024-05-03T10:19:56.690 INFO:teuthology.orchestra.run.smithi043.stderr:Generating a dashboard self-signed certificate... 2024-05-03T10:19:57.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:57 smithi043 ceph-mon[22404]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-03T10:19:57.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:57 smithi043 ceph-mon[22404]: mgrmap e11: y(active, since 1.05596s) 2024-05-03T10:19:57.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:57 smithi043 ceph-mon[22404]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-03T10:19:57.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:19:57.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:19:57.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:19:57.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi043", "caps": []}]: dispatch 2024-05-03T10:19:57.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi043", "caps": []}]': finished 2024-05-03T10:19:57.406 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: Self-signed certificate created 2024-05-03T10:19:57.980 INFO:teuthology.orchestra.run.smithi043.stderr:Creating initial admin user... 2024-05-03T10:19:58.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:58 smithi043 ceph-mon[22404]: Deploying daemon agent.smithi043 on smithi043 2024-05-03T10:19:58.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:58 smithi043 ceph-mon[22404]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:19:58.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:19:58.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:19:58.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:58 smithi043 ceph-mon[22404]: mgrmap e12: y(active, since 2s) 2024-05-03T10:19:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$XYYVPtxzJO9jwnuv5IARtex5UvMpJ7bEi8mhpftrmJLg01KSqvtnK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714731598, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-03T10:19:59.453 INFO:teuthology.orchestra.run.smithi043.stderr:Fetching dashboard port number... 2024-05-03T10:19:59.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:59 smithi043 ceph-mon[22404]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:19:59.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:19:59 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:00.137 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: 8443 2024-05-03T10:20:00.725 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:00 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3228601920' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-03T10:20:00.738 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: disabled 2024-05-03T10:20:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:systemctl: inactive 2024-05-03T10:20:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:firewalld.service is not enabled 2024-05-03T10:20:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-03T10:20:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:Ceph Dashboard is now available at: 2024-05-03T10:20:00.747 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:20:00.748 INFO:teuthology.orchestra.run.smithi043.stderr: URL: https://smithi043:8443/ 2024-05-03T10:20:00.748 INFO:teuthology.orchestra.run.smithi043.stderr: User: admin 2024-05-03T10:20:00.748 INFO:teuthology.orchestra.run.smithi043.stderr: Password: tnzl53omh2 2024-05-03T10:20:00.748 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:20:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:Enabling autotune for osd_memory_target 2024-05-03T10:20:02.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:02 smithi043 ceph-mon[22404]: mgrmap e13: y(active, since 6s) 2024-05-03T10:20:02.882 INFO:teuthology.orchestra.run.smithi043.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-05-03T10:20:03.569 INFO:teuthology.orchestra.run.smithi043.stderr:You can access the Ceph CLI with: 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:Please consider enabling telemetry to help improve Ceph: 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr: ceph telemetry on 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:For more information see: 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:20:03.570 INFO:teuthology.orchestra.run.smithi043.stderr:Bootstrap complete. 2024-05-03T10:20:03.765 INFO:tasks.cephadm:Fetching config... 2024-05-03T10:20:03.765 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:20:03.765 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-03T10:20:03.781 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-03T10:20:03.781 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:20:03.781 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-03T10:20:03.837 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-03T10:20:03.837 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:20:03.838 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-03T10:20:03.940 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-03T10:20:03.940 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:20:03.940 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-03T10:20:03.997 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-03T10:20:03.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsUfoskjRV0rkRbU46s/iepzEE36IPq9O8f6CR+kbJl8aWcCjb1EaZhMwoT1VlII5YuxSZr23tjAc+JocXBFpQS5iMdDHBjJfv7+ctbYhjhrbT9bucq4RGbsV/Da9JiYVvxMSwwIQp4sCB5ua/KgWmK3zMGVy8/cPpVi2gvDWXUEMfsCneTgrs/rBw/ZemfbfKjk23ZgHtEtD5uxIXpWidOjTz9e/nIuc1Y+XlsWdizuaTIYvHprEYmIXzXkL4dr9zKKdCno77w6orid2kOSN6xwg8P9RklXwXqBmbbdwof1DlXoAZXXdM7S6xADhYbDjG5yZg8ZnfdJbDUFzRlb4OkzkmPNct0dDJSOIJ4ohhYvpwsxDxfdSWEEakYIFiHJCrjgiy6R0ACNZtjQyb0PbEgTey8TMWQfumzUgl+tg9rT2q3ipoFckQLbz4Euoe33sKdmzXAYRnSs/FY/5EAH+nKQtw37oXNmOk5C5mx+KyWngIL1Pd7gLjGnS/r5xxqFU= ceph-79bfd44e-0936-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-03T10:20:04.071 INFO:teuthology.orchestra.run.smithi043.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsUfoskjRV0rkRbU46s/iepzEE36IPq9O8f6CR+kbJl8aWcCjb1EaZhMwoT1VlII5YuxSZr23tjAc+JocXBFpQS5iMdDHBjJfv7+ctbYhjhrbT9bucq4RGbsV/Da9JiYVvxMSwwIQp4sCB5ua/KgWmK3zMGVy8/cPpVi2gvDWXUEMfsCneTgrs/rBw/ZemfbfKjk23ZgHtEtD5uxIXpWidOjTz9e/nIuc1Y+XlsWdizuaTIYvHprEYmIXzXkL4dr9zKKdCno77w6orid2kOSN6xwg8P9RklXwXqBmbbdwof1DlXoAZXXdM7S6xADhYbDjG5yZg8ZnfdJbDUFzRlb4OkzkmPNct0dDJSOIJ4ohhYvpwsxDxfdSWEEakYIFiHJCrjgiy6R0ACNZtjQyb0PbEgTey8TMWQfumzUgl+tg9rT2q3ipoFckQLbz4Euoe33sKdmzXAYRnSs/FY/5EAH+nKQtw37oXNmOk5C5mx+KyWngIL1Pd7gLjGnS/r5xxqFU= ceph-79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:20:04.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsUfoskjRV0rkRbU46s/iepzEE36IPq9O8f6CR+kbJl8aWcCjb1EaZhMwoT1VlII5YuxSZr23tjAc+JocXBFpQS5iMdDHBjJfv7+ctbYhjhrbT9bucq4RGbsV/Da9JiYVvxMSwwIQp4sCB5ua/KgWmK3zMGVy8/cPpVi2gvDWXUEMfsCneTgrs/rBw/ZemfbfKjk23ZgHtEtD5uxIXpWidOjTz9e/nIuc1Y+XlsWdizuaTIYvHprEYmIXzXkL4dr9zKKdCno77w6orid2kOSN6xwg8P9RklXwXqBmbbdwof1DlXoAZXXdM7S6xADhYbDjG5yZg8ZnfdJbDUFzRlb4OkzkmPNct0dDJSOIJ4ohhYvpwsxDxfdSWEEakYIFiHJCrjgiy6R0ACNZtjQyb0PbEgTey8TMWQfumzUgl+tg9rT2q3ipoFckQLbz4Euoe33sKdmzXAYRnSs/FY/5EAH+nKQtw37oXNmOk5C5mx+KyWngIL1Pd7gLjGnS/r5xxqFU= ceph-79bfd44e-0936-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-03T10:20:04.123 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:03 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3410262511' entity='client.admin' 2024-05-03T10:20:04.126 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsUfoskjRV0rkRbU46s/iepzEE36IPq9O8f6CR+kbJl8aWcCjb1EaZhMwoT1VlII5YuxSZr23tjAc+JocXBFpQS5iMdDHBjJfv7+ctbYhjhrbT9bucq4RGbsV/Da9JiYVvxMSwwIQp4sCB5ua/KgWmK3zMGVy8/cPpVi2gvDWXUEMfsCneTgrs/rBw/ZemfbfKjk23ZgHtEtD5uxIXpWidOjTz9e/nIuc1Y+XlsWdizuaTIYvHprEYmIXzXkL4dr9zKKdCno77w6orid2kOSN6xwg8P9RklXwXqBmbbdwof1DlXoAZXXdM7S6xADhYbDjG5yZg8ZnfdJbDUFzRlb4OkzkmPNct0dDJSOIJ4ohhYvpwsxDxfdSWEEakYIFiHJCrjgiy6R0ACNZtjQyb0PbEgTey8TMWQfumzUgl+tg9rT2q3ipoFckQLbz4Euoe33sKdmzXAYRnSs/FY/5EAH+nKQtw37oXNmOk5C5mx+KyWngIL1Pd7gLjGnS/r5xxqFU= ceph-79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:20:04.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-03T10:20:04.378 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:04 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:20:04] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:05.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:05.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:05.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:05.244 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:05.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:05.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:06.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:08.520 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:08 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:20:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:09.186 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:08 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:20:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:09.290 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-03T10:20:09.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-03T10:20:09.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:09 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:09.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:09 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2416252705' entity='client.admin' 2024-05-03T10:20:09.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:09 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:09.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:09 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:09.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:09 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:09.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:09 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:12.013 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2024-05-03T10:20:12.013 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-03T10:20:12.014 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2024-05-03T10:20:12.035 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-03T10:20:12.035 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:20:12.095 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2024-05-03T10:20:12.095 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch host add smithi181 2024-05-03T10:20:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:12 smithi043 ceph-mon[22404]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:20:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:12 smithi043 ceph-mon[22404]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-03T10:20:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:12 smithi043 ceph-mon[22404]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:20:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:12.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:14.799 INFO:teuthology.orchestra.run.smithi043.stdout:Added host 'smithi181' with addr '172.21.15.181' 2024-05-03T10:20:15.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:14 smithi043 ceph-mon[22404]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:20:15.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:14 smithi043 ceph-mon[22404]: Deploying cephadm binary to smithi181 2024-05-03T10:20:15.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-03T10:20:15.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:15.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: Added host smithi181 2024-05-03T10:20:15.995 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi181", "caps": []}]: dispatch 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi181", "caps": []}]': finished 2024-05-03T10:20:15.996 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:15 smithi043 ceph-mon[22404]: Deploying daemon agent.smithi181 on smithi181 2024-05-03T10:20:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:20:17.341 INFO:teuthology.orchestra.run.smithi043.stdout:[{"addr": "172.21.15.43", "hostname": "smithi043", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2024-05-03T10:20:17.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:17 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:20:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: Updating smithi181:/etc/ceph/ceph.conf 2024-05-03T10:20:17.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:20:17.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:17.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:17.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:17.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:17.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:17.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:17.905 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:17 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:20:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:17.932 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-03T10:20:17.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd crush tunables default 2024-05-03T10:20:18.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:18 smithi043 ceph-mon[22404]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:18.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:18.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:18 smithi043 ceph-mon[22404]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T10:20:18.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:18.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:19.441 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:19 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:20:19] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:20.246 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2024-05-03T10:20:20.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:20 smithi043 ceph-mon[22404]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:20.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:20 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:20.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:20 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:20.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:20 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2640490711' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-03T10:20:20.517 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:20 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:20.830 INFO:tasks.cephadm:Adding mon.a on smithi043 2024-05-03T10:20:20.830 INFO:tasks.cephadm:Adding mon.c on smithi043 2024-05-03T10:20:20.830 INFO:tasks.cephadm:Adding mon.b on smithi181 2024-05-03T10:20:20.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch apply mon '3;smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi181:172.21.15.181=b' 2024-05-03T10:20:21.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:21 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2640490711' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-03T10:20:21.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:21 smithi043 ceph-mon[22404]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T10:20:22.578 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2024-05-03T10:20:23.009 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:22 smithi043 ceph-mon[22404]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:23.127 DEBUG:teuthology.orchestra.run.smithi043:mon.c> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.c.service 2024-05-03T10:20:23.129 DEBUG:teuthology.orchestra.run.smithi181:mon.b> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.b.service 2024-05-03T10:20:23.132 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T10:20:23.132 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T10:20:23.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi181:172.21.15.181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:20:23.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: Saving service mon spec with placement smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi181:172.21.15.181=b;count:3 2024-05-03T10:20:23.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:23.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:23.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:23.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:23.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:23.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:23.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:23.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:23 smithi043 ceph-mon[22404]: Deploying daemon mon.c on smithi043 2024-05-03T10:20:24.784 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:20:24.785 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"79bfd44e-0936-11ef-bc97-c7b262605968","modified":"2024-05-03T10:19:03.303907Z","created":"2024-05-03T10:19:03.303907Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-03T10:20:24.786 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-05-03T10:20:24.867 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:24 smithi043 ceph-mon[22404]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:25.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:25 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/1483442809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T10:20:26.339 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T10:20:26.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T10:20:26.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:26 smithi043 ceph-mon[22404]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:27.103 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:26 smithi043 podman[27212]: 2024-05-03 10:20:26.964474039 +0000 UTC m=+0.122185678 container create fe000a9cec3812e1a8f997a1e3fb9cc05752c1cb31dbddca10aaa06e8c408912 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-c, GIT_CLEAN=True, ceph=True, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vcs-type=git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, maintainer=Guillaume Abrioux , RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc.) 2024-05-03T10:20:27.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 podman[27212]: 2024-05-03 10:20:27.21754949 +0000 UTC m=+0.375261138 container init fe000a9cec3812e1a8f997a1e3fb9cc05752c1cb31dbddca10aaa06e8c408912 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-c, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, ceph=True, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, release=754) 2024-05-03T10:20:27.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 podman[27212]: 2024-05-03 10:20:27.220892776 +0000 UTC m=+0.378604414 container start fe000a9cec3812e1a8f997a1e3fb9cc05752c1cb31dbddca10aaa06e8c408912 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-c, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, release=754, distribution-scope=public, architecture=x86_64, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container) 2024-05-03T10:20:27.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: set uid:gid to 167:167 (ceph:ceph) 2024-05-03T10:20:27.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-03T10:20:27.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: pidfile_write: ignore empty --pid-file 2024-05-03T10:20:27.361 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: load: jerasure load: lrc 2024-05-03T10:20:27.362 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: RocksDB version: 6.15.5 2024-05-03T10:20:27.362 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-03T10:20:27.362 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Compile date Apr 18 2022 2024-05-03T10:20:27.362 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: DB SUMMARY 2024-05-03T10:20:27.362 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: DB Session ID: DB1OYIJ8T7MKLSPSCW5G 2024-05-03T10:20:27.362 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: CURRENT file: CURRENT 2024-05-03T10:20:27.362 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: IDENTITY file: IDENTITY 2024-05-03T10:20:27.362 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.error_if_exists: 0 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.create_if_missing: 0 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.paranoid_checks: 1 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.env: 0x556ca24c6860 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.fs: Posix File System 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.info_log: 0x556ca33b7dc0 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_file_opening_threads: 16 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.statistics: (nil) 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.use_fsync: 0 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_log_file_size: 0 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-03T10:20:27.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.keep_log_file_num: 1000 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.recycle_log_file_num: 0 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.allow_fallocate: 1 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.allow_mmap_reads: 0 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.allow_mmap_writes: 0 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.use_direct_reads: 0 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.create_missing_column_families: 0 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.db_log_dir: 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-03T10:20:27.364 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.advise_random_on_open: 1 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.db_write_buffer_size: 0 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.write_buffer_manager: 0x556ca34a82d0 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.rate_limiter: (nil) 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.wal_recovery_mode: 2 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.enable_thread_tracking: 0 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.enable_pipelined_write: 0 2024-05-03T10:20:27.365 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.unordered_write: 0 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.row_cache: None 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.wal_filter: None 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.allow_ingest_behind: 0 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.preserve_deletes: 0 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.two_write_queues: 0 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.manual_wal_flush: 0 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.atomic_flush: 0 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-03T10:20:27.366 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.log_readahead_size: 0 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.best_efforts_recovery: 0 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.allow_data_in_errors: 0 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.db_host_id: __hostname__ 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_background_jobs: 2 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_background_compactions: -1 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_subcompactions: 1 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-03T10:20:27.367 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_total_wal_size: 0 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_open_files: -1 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bytes_per_sync: 0 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_readahead_size: 0 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_background_flushes: -1 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Compression algorithms supported: 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: kZSTD supported: 0 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: kXpressCompression supported: 0 2024-05-03T10:20:27.368 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: kLZ4HCCompression supported: 1 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: kLZ4Compression supported: 1 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: kBZip2Compression supported: 0 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: kZlibCompression supported: 1 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: kSnappyCompression supported: 1 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.merge_operator: 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_filter: None 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_filter_factory: None 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.sst_partitioner_factory: None 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-03T10:20:27.369 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556ca3383d00) 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: cache_index_and_filter_blocks: 1 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: pin_top_level_index_and_filter: 1 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: index_type: 0 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: data_block_index_type: 0 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: index_shortening: 1 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: hash_index_allow_collision: 1 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: checksum: 1 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: no_block_cache: 0 2024-05-03T10:20:27.370 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache: 0x556ca33ee170 2024-05-03T10:20:27.371 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache_name: BinnedLRUCache 2024-05-03T10:20:27.371 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache_options: 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: capacity : 536870912 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: num_shard_bits : 4 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: strict_capacity_limit : 0 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: high_pri_pool_ratio: 0.000 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_cache_compressed: (nil) 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: persistent_cache: (nil) 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_size: 4096 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_size_deviation: 10 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_restart_interval: 16 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: index_block_restart_interval: 1 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: metadata_block_size: 4096 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: partition_filters: 0 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: use_delta_encoding: 1 2024-05-03T10:20:27.372 INFO:journalctl@ceph.mon.c.smithi043.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout: whole_key_filtering: 1 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout: verify_compression: 0 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout: read_amp_bytes_per_bit: 0 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout: format_version: 4 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout: enable_index_compression: 1 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout: block_align: 0 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.write_buffer_size: 33554432 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_write_buffer_number: 2 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compression: NoCompression 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bottommost_compression: Disabled 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.prefix_extractor: nullptr 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.num_levels: 7 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-03T10:20:27.373 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compression_opts.level: 32767 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compression_opts.strategy: 0 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-03T10:20:27.374 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compression_opts.enabled: false 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.target_file_size_base: 67108864 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-03T10:20:27.375 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.arena_block_size: 4194304 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.disable_auto_compactions: 0 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-03T10:20:27.376 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.table_properties_collectors: 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.inplace_update_support: 0 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.bloom_locality: 0 2024-05-03T10:20:27.377 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.max_successive_merges: 0 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.paranoid_file_checks: 0 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.force_consistency_checks: 1 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.report_bg_io_stats: 0 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.ttl: 2592000 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.enable_blob_files: false 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.min_blob_size: 0 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.blob_file_size: 268435456 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-03T10:20:27.378 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714731627246645, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714731627246946, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1714731627, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "03db69b8-c9b7-4edc-a1da-cecf8efca902", "db_session_id": "DB1OYIJ8T7MKLSPSCW5G"}} 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714731627247277, "job": 1, "event": "recovery_finished"} 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x556ca33d4a80 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: DB pointer 0x556ca33e4000 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** DB Stats ** 2024-05-03T10:20:27.379 INFO:journalctl@ceph.mon.c.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-03T10:20:27.380 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T10:20:27.381 INFO:journalctl@ceph.mon.c.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** Compaction Stats [default] ** 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-03T10:20:27.382 INFO:journalctl@ceph.mon.c.smithi043.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-03T10:20:27.383 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: using public_addrv [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: starting mon.c rank -1 at public addrs [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] at bind addrs [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(???) e0 preinit fsid 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).mds e1 new map 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout: e1 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout: legacy client fscid: -1 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout: 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout: No filesystems configured 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-03T10:20:27.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/1483442809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 ceph-mon[27243]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 bash[27212]: fe000a9cec3812e1a8f997a1e3fb9cc05752c1cb31dbddca10aaa06e8c408912 2024-05-03T10:20:27.385 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:27 smithi043 systemd[1]: Started Ceph mon.c for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:20:32.543 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:32 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:20:32] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:32.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:32.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:20:32.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: mon.a calling monitor election 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: mon.c calling monitor election 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: monmap e2: 2 mons at {a=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],c=[v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0]} 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: fsmap 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: mgrmap e13: y(active, since 37s) 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: overall HEALTH_OK 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:32.545 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: mon.a calling monitor election 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: mon.c calling monitor election 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: monmap e2: 2 mons at {a=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],c=[v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0]} 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: fsmap 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: mgrmap e13: y(active, since 37s) 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: overall HEALTH_OK 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:32.547 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:32.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:32.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:32.548 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:32.603 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:20:32.603 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"79bfd44e-0936-11ef-bc97-c7b262605968","modified":"2024-05-03T10:20:27.265965Z","created":"2024-05-03T10:19:03.303907Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-03T10:20:32.604 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-05-03T10:20:33.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:33 smithi043 ceph-mon[22404]: Deploying daemon mon.b on smithi181 2024-05-03T10:20:33.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:33 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/2574955393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T10:20:33.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:33 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:33.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:33 smithi043 ceph-mon[27243]: Deploying daemon mon.b on smithi181 2024-05-03T10:20:33.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:33 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/2574955393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T10:20:33.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:33 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:34.752 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T10:20:34.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T10:20:34.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:34 smithi043 ceph-mon[22404]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:34.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:34 smithi043 ceph-mon[27243]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:36.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:36 smithi043 ceph-mon[22404]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:36.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:36 smithi043 ceph-mon[27243]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:37.085 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:20:37.085 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"79bfd44e-0936-11ef-bc97-c7b262605968","modified":"2024-05-03T10:20:27.265965Z","created":"2024-05-03T10:19:03.303907Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-03T10:20:37.086 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-05-03T10:20:37.171 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:37 smithi181 ceph-mon[23393]: mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-05-03T10:20:37.483 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:37 smithi181 ceph-mon[23393]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-05-03T10:20:37.483 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:37 smithi181 bash[23317]: 5e27fbb569074b97b3d34f6be003c51f311547ad8a1114e57b3182984896ca46 2024-05-03T10:20:37.483 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:37 smithi181 systemd[1]: Started Ceph mon.b for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:20:39.494 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T10:20:39.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T10:20:42.457 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:42 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:20:42] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:42.457 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:42 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:20:42] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: mon.a calling monitor election 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: mon.c calling monitor election 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:42.458 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: monmap e3: 3 mons at {a=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0]} 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: fsmap 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: mgrmap e13: y(active, since 47s) 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: mon.b (rank 2) addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] is down (out of quorum) 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:42.459 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: mon.a calling monitor election 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: mon.c calling monitor election 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.460 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: monmap e3: 3 mons at {a=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0]} 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: fsmap 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: mgrmap e13: y(active, since 47s) 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: mon.b (rank 2) addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] is down (out of quorum) 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:42.461 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:42.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:42.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:42.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:42.462 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:42.591 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-05-03T10:20:42.591 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":3,"fsid":"79bfd44e-0936-11ef-bc97-c7b262605968","modified":"2024-05-03T10:20:37.174254Z","created":"2024-05-03T10:19:03.303907Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-03T10:20:42.593 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 3 2024-05-03T10:20:43.175 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-03T10:20:43.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph config generate-minimal-conf 2024-05-03T10:20:43.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:20:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:43.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-03T10:20:43.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: Updating smithi181:/etc/ceph/ceph.conf 2024-05-03T10:20:43.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: Reconfiguring mon.a (unknown last config time)... 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: Reconfiguring daemon mon.a on smithi043 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/754672790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-03T10:20:43.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: Updating smithi181:/etc/ceph/ceph.conf 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: Reconfiguring mon.a (unknown last config time)... 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: Reconfiguring daemon mon.a on smithi043 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/754672790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:43.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mon.a calling monitor election 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mon.c calling monitor election 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:44.292 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: monmap e3: 3 mons at {a=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0]} 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: fsmap 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mgrmap e13: y(active, since 47s) 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mon.b (rank 2) addr [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] is down (out of quorum) 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:44.293 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: Updating smithi181:/etc/ceph/ceph.conf 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: Reconfiguring mon.a (unknown last config time)... 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T10:20:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: Reconfiguring daemon mon.a on smithi043 2024-05-03T10:20:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/754672790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T10:20:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.295 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:44.295 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:20:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:45.287 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mon.b calling monitor election 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mon.b calling monitor election 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mon.c calling monitor election 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mon.a calling monitor election 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: monmap e3: 3 mons at {a=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0]} 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: fsmap 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: mgrmap e13: y(active, since 49s) 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: Cluster is now healthy 2024-05-03T10:20:45.288 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:44 smithi181 ceph-mon[23393]: overall HEALTH_OK 2024-05-03T10:20:45.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: mon.b calling monitor election 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: mon.b calling monitor election 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: mon.c calling monitor election 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: mon.a calling monitor election 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: monmap e3: 3 mons at {a=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0]} 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: fsmap 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: mgrmap e13: y(active, since 49s) 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: Cluster is now healthy 2024-05-03T10:20:45.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[22404]: overall HEALTH_OK 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: mon.b calling monitor election 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: mon.b calling monitor election 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: mon.c calling monitor election 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: mon.a calling monitor election 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: monmap e3: 3 mons at {a=[v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0],b=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0],c=[v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0]} 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: fsmap 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: mgrmap e13: y(active, since 49s) 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: Cluster is now healthy 2024-05-03T10:20:45.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:44 smithi043 ceph-mon[27243]: overall HEALTH_OK 2024-05-03T10:20:46.082 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:45 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:20:45] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:46.082 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[22404]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:46.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:46 smithi181 ceph-mon[23393]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:46.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:46.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:46.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T10:20:46.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:46.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:46.400 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[27243]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T10:20:46.401 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:46.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:46.402 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:47.028 INFO:teuthology.orchestra.run.smithi043.stdout:# minimal ceph.conf for 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:20:47.029 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-05-03T10:20:47.029 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:20:47.029 INFO:teuthology.orchestra.run.smithi043.stdout: mon_host = [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-05-03T10:20:47.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:47 smithi181 ceph-mon[23393]: Reconfiguring mon.c (monmap changed)... 2024-05-03T10:20:47.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:47 smithi181 ceph-mon[23393]: Reconfiguring daemon mon.c on smithi043 2024-05-03T10:20:47.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:47 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/82071145' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:47.889 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:47 smithi043 ceph-mon[22404]: Reconfiguring mon.c (monmap changed)... 2024-05-03T10:20:47.889 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:47 smithi043 ceph-mon[22404]: Reconfiguring daemon mon.c on smithi043 2024-05-03T10:20:47.889 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:47 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/82071145' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:47.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:47 smithi043 ceph-mon[27243]: Reconfiguring mon.c (monmap changed)... 2024-05-03T10:20:47.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:47 smithi043 ceph-mon[27243]: Reconfiguring daemon mon.c on smithi043 2024-05-03T10:20:47.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:47 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/82071145' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:47.916 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-03T10:20:47.917 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:20:47.917 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.conf 2024-05-03T10:20:47.948 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:20:47.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:20:48.013 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-03T10:20:48.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2024-05-03T10:20:48.047 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-03T10:20:48.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:20:48.115 INFO:tasks.cephadm:Adding mgr.y on smithi043 2024-05-03T10:20:48.115 INFO:tasks.cephadm:Adding mgr.x on smithi181 2024-05-03T10:20:48.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch apply mgr '2;smithi043=y;smithi181=x' 2024-05-03T10:20:48.677 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:48 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:20:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:48.677 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:48 smithi043 ceph-mon[22404]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:48.678 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:48.678 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:48.678 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:48 smithi043 ceph-mon[27243]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:48.678 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:48.678 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:48.767 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:48 smithi181 ceph-mon[23393]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:48.768 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:48.768 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:50.018 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mgr update... 2024-05-03T10:20:50.628 DEBUG:teuthology.orchestra.run.smithi181:mgr.x> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mgr.x.service 2024-05-03T10:20:50.633 INFO:tasks.cephadm:Deploying OSDs... 2024-05-03T10:20:50.633 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:20:50.633 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-05-03T10:20:50.653 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-03T10:20:50.654 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-05-03T10:20:50.712 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-03T10:20:50.712 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T10:20:50.712 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-03T10:20:50.712 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T10:20:50.713 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T10:20:50.713 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-05-03 10:20:47.133490295 +0000 2024-05-03T10:20:50.713 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-05-03 10:18:05.319682667 +0000 2024-05-03T10:20:50.713 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-05-03 10:18:05.319682667 +0000 2024-05-03T10:20:50.713 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-05-03 10:18:05.319682667 +0000 2024-05-03T10:20:50.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-03T10:20:50.782 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-05-03T10:20:50.783 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-05-03T10:20:50.783 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000131894 s, 3.9 MB/s 2024-05-03T10:20:50.784 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-03T10:20:50.843 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-05-03T10:20:50.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:50 smithi181 ceph-mon[23393]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:50.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:50 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:50.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:50 smithi181 ceph-mon[23393]: Reconfiguring mon.b (monmap changed)... 2024-05-03T10:20:50.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:50 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:50.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:50 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T10:20:50.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:50 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:50.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:50 smithi181 ceph-mon[23393]: Reconfiguring daemon mon.b on smithi181 2024-05-03T10:20:50.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:50 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-03T10:20:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T10:20:50.902 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-03T10:20:50.902 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T10:20:50.902 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T10:20:50.902 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-05-03 10:20:47.133490295 +0000 2024-05-03T10:20:50.902 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-05-03 10:18:05.661678696 +0000 2024-05-03T10:20:50.902 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-05-03 10:18:05.661678696 +0000 2024-05-03T10:20:50.902 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-05-03 10:18:05.661678696 +0000 2024-05-03T10:20:50.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-03T10:20:50.967 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[22404]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:50.967 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:50.967 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[22404]: Reconfiguring mon.b (monmap changed)... 2024-05-03T10:20:50.967 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[22404]: Reconfiguring daemon mon.b on smithi181 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[27243]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[27243]: Reconfiguring mon.b (monmap changed)... 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:50.968 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[27243]: Reconfiguring daemon mon.b on smithi181 2024-05-03T10:20:50.969 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:50 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:50.969 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-05-03T10:20:50.969 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-05-03T10:20:50.969 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000129148 s, 4.0 MB/s 2024-05-03T10:20:50.971 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-03T10:20:51.029 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-05-03T10:20:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-03T10:20:51.087 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T10:20:51.087 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-03T10:20:51.087 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T10:20:51.088 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T10:20:51.088 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-05-03 10:20:47.133490295 +0000 2024-05-03T10:20:51.088 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-05-03 10:18:06.012674620 +0000 2024-05-03T10:20:51.088 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-05-03 10:18:06.012674620 +0000 2024-05-03T10:20:51.088 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-05-03 10:18:06.012674620 +0000 2024-05-03T10:20:51.088 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-03T10:20:51.153 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-05-03T10:20:51.153 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-05-03T10:20:51.153 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000119939 s, 4.3 MB/s 2024-05-03T10:20:51.155 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-03T10:20:51.213 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-05-03T10:20:51.270 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-03T10:20:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T10:20:51.271 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-03T10:20:51.271 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T10:20:51.271 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T10:20:51.271 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-05-03 10:20:47.133490295 +0000 2024-05-03T10:20:51.271 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-05-03 10:18:06.348670718 +0000 2024-05-03T10:20:51.271 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-05-03 10:18:06.348670718 +0000 2024-05-03T10:20:51.271 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: 2024-05-03 10:18:06.348670718 +0000 2024-05-03T10:20:51.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-03T10:20:51.336 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-05-03T10:20:51.336 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-05-03T10:20:51.336 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000118556 s, 4.3 MB/s 2024-05-03T10:20:51.337 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-03T10:20:51.395 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-03T10:20:51.395 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-05-03T10:20:51.411 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-03T10:20:51.411 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-05-03T10:20:51.468 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-03T10:20:51.468 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T10:20:51.468 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-05-03T10:20:51.469 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T10:20:51.469 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T10:20:51.469 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-05-03 10:20:45.252191411 +0000 2024-05-03T10:20:51.469 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-05-03 10:17:59.463127200 +0000 2024-05-03T10:20:51.469 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-05-03 10:17:59.463127200 +0000 2024-05-03T10:20:51.469 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-05-03 10:17:59.463127200 +0000 2024-05-03T10:20:51.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-03T10:20:51.531 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-05-03T10:20:51.531 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-05-03T10:20:51.531 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000209808 s, 2.4 MB/s 2024-05-03T10:20:51.532 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-03T10:20:51.588 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-05-03T10:20:51.643 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-03T10:20:51.643 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T10:20:51.644 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-03T10:20:51.644 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T10:20:51.644 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T10:20:51.644 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-05-03 10:20:45.252191411 +0000 2024-05-03T10:20:51.644 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-05-03 10:17:59.793120469 +0000 2024-05-03T10:20:51.644 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-05-03 10:17:59.793120469 +0000 2024-05-03T10:20:51.644 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-05-03 10:17:59.793120469 +0000 2024-05-03T10:20:51.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi043=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: Saving service mgr spec with placement smithi043=y;smithi181=x;count:2 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T10:20:51.705 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T10:20:51.706 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:51.707 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-05-03T10:20:51.707 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-05-03T10:20:51.708 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.00020123 s, 2.5 MB/s 2024-05-03T10:20:51.709 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-03T10:20:51.765 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-05-03T10:20:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-03T10:20:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T10:20:51.821 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-03T10:20:51.821 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T10:20:51.821 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T10:20:51.821 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-05-03 10:20:45.252191411 +0000 2024-05-03T10:20:51.821 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-05-03 10:18:00.141113372 +0000 2024-05-03T10:20:51.821 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-05-03 10:18:00.141113372 +0000 2024-05-03T10:20:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-05-03 10:18:00.141113372 +0000 2024-05-03T10:20:51.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-03T10:20:51.882 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-05-03T10:20:51.882 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-05-03T10:20:51.882 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.00022338 s, 2.3 MB/s 2024-05-03T10:20:51.883 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-03T10:20:51.940 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-05-03T10:20:51.995 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-03T10:20:51.995 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T10:20:51.995 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-05-03T10:20:51.996 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T10:20:51.996 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T10:20:51.996 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-05-03 10:20:45.252191411 +0000 2024-05-03T10:20:51.996 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-05-03 10:18:00.463106803 +0000 2024-05-03T10:20:51.996 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-05-03 10:18:00.463106803 +0000 2024-05-03T10:20:51.996 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-05-03 10:18:00.463106803 +0000 2024-05-03T10:20:51.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-03T10:20:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi043=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:20:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: Saving service mgr spec with placement smithi043=y;smithi181=x;count:2 2024-05-03T10:20:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T10:20:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi043=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: Saving service mgr spec with placement smithi043=y;smithi181=x;count:2 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T10:20:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T10:20:52.044 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:52.058 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-05-03T10:20:52.058 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-05-03T10:20:52.058 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000169666 s, 3.0 MB/s 2024-05-03T10:20:52.059 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-03T10:20:52.115 INFO:tasks.cephadm:Deploying osd.0 on smithi043 with /dev/vg_nvme/lv_4... 2024-05-03T10:20:52.115 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-03T10:20:52.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:52 smithi043 ceph-mon[27243]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:52.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:52 smithi043 ceph-mon[27243]: Deploying daemon mgr.x on smithi181 2024-05-03T10:20:52.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:52 smithi043 ceph-mon[22404]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:52.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:52 smithi043 ceph-mon[22404]: Deploying daemon mgr.x on smithi181 2024-05-03T10:20:52.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:52 smithi181 ceph-mon[23393]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:52.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:52 smithi181 ceph-mon[23393]: Deploying daemon mgr.x on smithi181 2024-05-03T10:20:53.251 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 systemd[1]: Starting Ceph mgr.x for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:20:53.510 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 podman[24890]: 2024-05-03 10:20:53.15383794 +0000 UTC m=+0.018648515 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-03T10:20:53.510 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 podman[24890]: 2024-05-03 10:20:53.273836693 +0000 UTC m=+0.138647267 container create f42a19b78d32214ec5751946c8c0cb07c8c10135179ec9e3cb3cf870ab41a9c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, maintainer=Guillaume Abrioux , name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8) 2024-05-03T10:20:53.511 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 podman[24890]: 2024-05-03 10:20:53.509530446 +0000 UTC m=+0.374341017 container init f42a19b78d32214ec5751946c8c0cb07c8c10135179ec9e3cb3cf870ab41a9c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, name=centos-stream, RELEASE=HEAD, release=754, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, architecture=x86_64, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.component=centos-stream-container, distribution-scope=public, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-03T10:20:53.787 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 podman[24890]: 2024-05-03 10:20:53.51261232 +0000 UTC m=+0.377422892 container start f42a19b78d32214ec5751946c8c0cb07c8c10135179ec9e3cb3cf870ab41a9c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x, name=centos-stream, distribution-scope=public, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-type=git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-03T10:20:53.787 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 ceph-mgr[24909]: -- 172.21.15.181:0/4001405998 <== mon.2 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55878624e340 con 0x558786227000 2024-05-03T10:20:53.787 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 bash[24890]: f42a19b78d32214ec5751946c8c0cb07c8c10135179ec9e3cb3cf870ab41a9c4 2024-05-03T10:20:53.787 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 systemd[1]: Started Ceph mgr.x for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:20:53.787 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:53.654+0000 7fabcacf3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T10:20:53.787 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:53.786+0000 7fabcacf3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:20:54.114 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-03T10:20:54.114 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-03T10:20:54.114 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-05-03T10:20:54.114 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-05-03T10:20:54.114 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00845822 s, 1.2 GB/s 2024-05-03T10:20:54.115 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-05-03T10:20:54.119 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:53 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:53.882+0000 7fabcacf3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T10:20:54.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:54 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:54.187+0000 7fabcacf3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T10:20:54.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:54 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:54.268+0000 7fabcacf3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T10:20:54.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:54.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:54.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:54.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:54.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:54.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: Reconfiguring mgr.y (unknown last config time)... 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[27243]: Reconfiguring daemon mgr.y on smithi043 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:54.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:54.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:20:54.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: Reconfiguring mgr.y (unknown last config time)... 2024-05-03T10:20:54.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T10:20:54.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T10:20:54.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:54.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:54 smithi043 ceph-mon[22404]: Reconfiguring daemon mgr.y on smithi043 2024-05-03T10:20:54.837 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_4 2024-05-03T10:20:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:55.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:55.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:20:55.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: Reconfiguring mgr.y (unknown last config time)... 2024-05-03T10:20:55.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T10:20:55.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T10:20:55.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:55.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:54 smithi181 ceph-mon[23393]: Reconfiguring daemon mgr.y on smithi043 2024-05-03T10:20:55.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:55 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:55.401+0000 7fabcacf3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T10:20:55.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:55 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:55.565+0000 7fabcacf3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T10:20:55.903 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:55 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:55.650+0000 7fabcacf3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T10:20:55.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:20:55.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:20:55.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:55.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:55.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:55.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:55.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:55.904 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:56.028 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:20:56.028 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:56.029 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:56.030 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:56.030 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:56.030 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:56.160 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:55 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:55.977+0000 7fabcacf3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T10:20:56.161 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:56 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:56.067+0000 7fabcacf3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T10:20:56.161 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:56 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:56.158+0000 7fabcacf3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T10:20:56.679 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:56 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:56.412+0000 7fabcacf3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:20:56.748 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:56 smithi043 ceph-mon[22404]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:56.748 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:56 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:20:56.748 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:56 smithi043 ceph-mon[27243]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:56.748 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:56 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:20:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:56 smithi181 ceph-mon[23393]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:56 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:20:57.119 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:56 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:56.873+0000 7fabcacf3000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T10:20:57.700 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:57 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:57.438+0000 7fabcacf3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T10:20:57.700 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:57 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:57.536+0000 7fabcacf3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T10:20:57.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:20:57.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:20:57.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:20:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:20:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:58.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:20:58.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:20:58.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:58.053 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:57 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:57.836+0000 7fabcacf3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T10:20:58.053 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:57 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:57.924+0000 7fabcacf3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T10:20:58.146 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:58 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:20:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:58.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:20:58 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:20:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:20:58.569 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:58 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:58.341+0000 7fabcacf3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T10:20:58.569 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:58 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:58.476+0000 7fabcacf3000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T10:20:58.569 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:58 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:58.567+0000 7fabcacf3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T10:20:58.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:58 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:58.679+0000 7fabcacf3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T10:20:58.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:58 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:58.768+0000 7fabcacf3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T10:20:58.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:58.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:20:58.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:58.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:58.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:58.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:58.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:20:58.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.954 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.955 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:20:58.956 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:20:58.956 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:20:58.956 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.956 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:58.956 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:20:59.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:59 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:59.310+0000 7fabcacf3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T10:21:00.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[22404]: Standby manager daemon x started 2024-05-03T10:21:00.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[22404]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-03T10:21:00.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[22404]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T10:21:00.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[22404]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-03T10:21:00.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[22404]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T10:21:00.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[27243]: Standby manager daemon x started 2024-05-03T10:21:00.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[27243]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-03T10:21:00.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[27243]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T10:21:00.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[27243]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-03T10:21:00.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:20:59 smithi043 ceph-mon[27243]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T10:21:00.119 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:20:59 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:20:59.672+0000 7fabcacf3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T10:21:00.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:59 smithi181 ceph-mon[23393]: Standby manager daemon x started 2024-05-03T10:21:00.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:59 smithi181 ceph-mon[23393]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-03T10:21:00.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:59 smithi181 ceph-mon[23393]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T10:21:00.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:59 smithi181 ceph-mon[23393]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-03T10:21:00.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:20:59 smithi181 ceph-mon[23393]: from='mgr.? 172.21.15.181:0/86479040' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T10:21:01.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[22404]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[22404]: mgrmap e14: y(active, since 64s), standbys: x 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1938665790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db4ef97-76b0-4df2-9b32-106b07e97bf2"}]: dispatch 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1938665790' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6db4ef97-76b0-4df2-9b32-106b07e97bf2"}]': finished 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[22404]: osdmap e5: 1 total, 0 up, 1 in 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3158955518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[27243]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[27243]: mgrmap e14: y(active, since 64s), standbys: x 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1938665790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db4ef97-76b0-4df2-9b32-106b07e97bf2"}]: dispatch 2024-05-03T10:21:01.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1938665790' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6db4ef97-76b0-4df2-9b32-106b07e97bf2"}]': finished 2024-05-03T10:21:01.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[27243]: osdmap e5: 1 total, 0 up, 1 in 2024-05-03T10:21:01.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:01.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:00 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3158955518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:21:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:00 smithi181 ceph-mon[23393]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:00 smithi181 ceph-mon[23393]: mgrmap e14: y(active, since 64s), standbys: x 2024-05-03T10:21:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:00 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T10:21:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:00 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1938665790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db4ef97-76b0-4df2-9b32-106b07e97bf2"}]: dispatch 2024-05-03T10:21:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:00 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1938665790' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6db4ef97-76b0-4df2-9b32-106b07e97bf2"}]': finished 2024-05-03T10:21:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:00 smithi181 ceph-mon[23393]: osdmap e5: 1 total, 0 up, 1 in 2024-05-03T10:21:01.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:00 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:01.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:00 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3158955518' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:21:02.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:02 smithi043 ceph-mon[27243]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:02.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:02 smithi043 ceph-mon[22404]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:02 smithi181 ceph-mon[23393]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:04.726 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:04 smithi043 ceph-mon[22404]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:04.727 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:04 smithi043 ceph-mon[27243]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:05.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:04 smithi181 ceph-mon[23393]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:06.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:06 smithi043 ceph-mon[22404]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:06.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-03T10:21:06.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:06.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:06 smithi043 ceph-mon[27243]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:06.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-03T10:21:06.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:07.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:06 smithi181 ceph-mon[23393]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:07.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-03T10:21:07.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:07.892 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:21:07 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:21:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:21:07.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:07 smithi043 ceph-mon[22404]: Deploying daemon osd.0 on smithi043 2024-05-03T10:21:07.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:07.893 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:07.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:07 smithi043 ceph-mon[27243]: Deploying daemon osd.0 on smithi043 2024-05-03T10:21:07.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:07.893 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:08.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:07 smithi181 ceph-mon[23393]: Deploying daemon osd.0 on smithi043 2024-05-03T10:21:08.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:08.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:09.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:08 smithi043 ceph-mon[22404]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:09.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:08 smithi043 ceph-mon[27243]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:08 smithi181 ceph-mon[23393]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:10.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:10 smithi043 ceph-mon[27243]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:10.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:10 smithi043 ceph-mon[22404]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:10 smithi181 ceph-mon[23393]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:13.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:12 smithi043 ceph-mon[22404]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:13.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:12 smithi043 ceph-mon[27243]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:12 smithi181 ceph-mon[23393]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:14 smithi181 ceph-mon[23393]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:14.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[22404]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:14.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:14.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:14.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:14.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:14.648 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:14.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[27243]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:14.648 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:14.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:14.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:14.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:14.649 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.105 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 0 on host 'smithi043' 2024-05-03T10:21:16.367 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:16 smithi181 ceph-mon[23393]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:16.367 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:16 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.367 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:16 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:16.367 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:16 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:16.367 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:16 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:16.367 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:16 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:21:16.367 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:16 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.367 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:16 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[27243]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:16.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[22404]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:21:16.680 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.681 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:16 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:16.790 DEBUG:teuthology.orchestra.run.smithi043:osd.0> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.0.service 2024-05-03T10:21:16.793 INFO:tasks.cephadm:Deploying osd.1 on smithi043 with /dev/vg_nvme/lv_3... 2024-05-03T10:21:16.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-03T10:21:17.262 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:21:17 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:21:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:21:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[22404]: Adjusting osd_memory_target on smithi043 to 15045M 2024-05-03T10:21:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[22404]: from='osd.0 [v2:172.21.15.43:6802/2876312201,v1:172.21.15.43:6803/2876312201]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T10:21:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[22404]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T10:21:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:17.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:17.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[27243]: Adjusting osd_memory_target on smithi043 to 15045M 2024-05-03T10:21:17.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[27243]: from='osd.0 [v2:172.21.15.43:6802/2876312201,v1:172.21.15.43:6803/2876312201]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T10:21:17.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[27243]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T10:21:17.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:17.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:17.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:17 smithi181 ceph-mon[23393]: Adjusting osd_memory_target on smithi043 to 15045M 2024-05-03T10:21:17.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:17 smithi181 ceph-mon[23393]: from='osd.0 [v2:172.21.15.43:6802/2876312201,v1:172.21.15.43:6803/2876312201]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T10:21:17.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:17 smithi181 ceph-mon[23393]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T10:21:17.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:17.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:18.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:18 smithi181 ceph-mon[23393]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:18.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:18 smithi181 ceph-mon[23393]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-03T10:21:18.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:18 smithi181 ceph-mon[23393]: osdmap e6: 1 total, 0 up, 1 in 2024-05-03T10:21:18.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:18.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:18 smithi181 ceph-mon[23393]: from='osd.0 [v2:172.21.15.43:6802/2876312201,v1:172.21.15.43:6803/2876312201]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:21:18.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:18 smithi181 ceph-mon[23393]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:21:18.757 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-03T10:21:18.757 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-03T10:21:18.757 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-05-03T10:21:18.757 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-05-03T10:21:18.757 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00995591 s, 1.1 GB/s 2024-05-03T10:21:18.758 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-05-03T10:21:18.758 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[27243]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:18.758 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[27243]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-03T10:21:18.758 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[27243]: osdmap e6: 1 total, 0 up, 1 in 2024-05-03T10:21:18.758 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:18.759 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[27243]: from='osd.0 [v2:172.21.15.43:6802/2876312201,v1:172.21.15.43:6803/2876312201]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:21:18.759 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[27243]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:21:18.759 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[22404]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:18.759 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[22404]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-03T10:21:18.759 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[22404]: osdmap e6: 1 total, 0 up, 1 in 2024-05-03T10:21:18.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:18.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[22404]: from='osd.0 [v2:172.21.15.43:6802/2876312201,v1:172.21.15.43:6803/2876312201]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:21:18.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:18 smithi043 ceph-mon[22404]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:21:18.760 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:21:18 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0[30418]: 2024-05-03T10:21:18.355+0000 7f3980233700 -1 osd.0 0 waiting for initial osdmap 2024-05-03T10:21:18.760 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:21:18 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0[30418]: 2024-05-03T10:21:18.358+0000 7f397b3cb700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T10:21:19.391 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_3 2024-05-03T10:21:19.410 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[22404]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:21:19.411 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[22404]: osdmap e7: 1 total, 0 up, 1 in 2024-05-03T10:21:19.411 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:19.411 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:19.411 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:19.411 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[27243]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:21:19.411 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[27243]: osdmap e7: 1 total, 0 up, 1 in 2024-05-03T10:21:19.411 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:19.411 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:19.411 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:19.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:19 smithi181 ceph-mon[23393]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:21:19.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:19 smithi181 ceph-mon[23393]: osdmap e7: 1 total, 0 up, 1 in 2024-05-03T10:21:19.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:19.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:19.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:20.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:20 smithi181 ceph-mon[23393]: purged_snaps scrub starts 2024-05-03T10:21:20.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:20 smithi181 ceph-mon[23393]: purged_snaps scrub ok 2024-05-03T10:21:20.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:20 smithi181 ceph-mon[23393]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:20.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:20 smithi181 ceph-mon[23393]: osd.0 [v2:172.21.15.43:6802/2876312201,v1:172.21.15.43:6803/2876312201] boot 2024-05-03T10:21:20.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:20 smithi181 ceph-mon[23393]: osdmap e8: 1 total, 1 up, 1 in 2024-05-03T10:21:20.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:20 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:20.678 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[22404]: purged_snaps scrub starts 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[22404]: purged_snaps scrub ok 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[22404]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[22404]: osd.0 [v2:172.21.15.43:6802/2876312201,v1:172.21.15.43:6803/2876312201] boot 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[22404]: osdmap e8: 1 total, 1 up, 1 in 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[27243]: purged_snaps scrub starts 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[27243]: purged_snaps scrub ok 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[27243]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[27243]: osd.0 [v2:172.21.15.43:6802/2876312201,v1:172.21.15.43:6803/2876312201] boot 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[27243]: osdmap e8: 1 total, 1 up, 1 in 2024-05-03T10:21:20.679 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:20 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:21:21.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:21 smithi043 ceph-mon[22404]: osdmap e9: 1 total, 1 up, 1 in 2024-05-03T10:21:21.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:21 smithi043 ceph-mon[27243]: osdmap e9: 1 total, 1 up, 1 in 2024-05-03T10:21:21.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:21 smithi181 ceph-mon[23393]: osdmap e9: 1 total, 1 up, 1 in 2024-05-03T10:21:22.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[22404]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:22.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[22404]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:21:22.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:21:22.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:21:22.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:22.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[27243]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:22.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[27243]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:21:22.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:21:22.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:21:22.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:22 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:22.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:22 smithi181 ceph-mon[23393]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:22.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:22 smithi181 ceph-mon[23393]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:21:22.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:22 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:21:22.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:22 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:21:22.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:22 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:23.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:21:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:21:23] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:21:24.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[22404]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:24.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:24.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[22404]: Detected new or changed devices on smithi043 2024-05-03T10:21:24.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:24.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1079487706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195e4d8e-8690-4122-845e-78c7ec670b64"}]: dispatch 2024-05-03T10:21:24.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195e4d8e-8690-4122-845e-78c7ec670b64"}]: dispatch 2024-05-03T10:21:24.649 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "195e4d8e-8690-4122-845e-78c7ec670b64"}]': finished 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[22404]: osdmap e10: 2 total, 1 up, 2 in 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[27243]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[27243]: Detected new or changed devices on smithi043 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1079487706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195e4d8e-8690-4122-845e-78c7ec670b64"}]: dispatch 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195e4d8e-8690-4122-845e-78c7ec670b64"}]: dispatch 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "195e4d8e-8690-4122-845e-78c7ec670b64"}]': finished 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[27243]: osdmap e10: 2 total, 1 up, 2 in 2024-05-03T10:21:24.650 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:24 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:24.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:24 smithi181 ceph-mon[23393]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:24.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:24 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:24.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:24 smithi181 ceph-mon[23393]: Detected new or changed devices on smithi043 2024-05-03T10:21:24.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:24 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:24.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:24 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1079487706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195e4d8e-8690-4122-845e-78c7ec670b64"}]: dispatch 2024-05-03T10:21:24.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:24 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195e4d8e-8690-4122-845e-78c7ec670b64"}]: dispatch 2024-05-03T10:21:24.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:24 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "195e4d8e-8690-4122-845e-78c7ec670b64"}]': finished 2024-05-03T10:21:24.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:24 smithi181 ceph-mon[23393]: osdmap e10: 2 total, 1 up, 2 in 2024-05-03T10:21:24.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:24 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:25.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:25 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/228823908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:21:25.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:25 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/228823908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:21:25.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:25 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/228823908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:21:26.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:26 smithi043 ceph-mon[22404]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:26.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:26 smithi043 ceph-mon[27243]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:26.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:26 smithi181 ceph-mon[23393]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:28.733 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:28 smithi043 ceph-mon[22404]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:28.734 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:28 smithi043 ceph-mon[27243]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:28.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:28 smithi181 ceph-mon[23393]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:30.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:30 smithi181 ceph-mon[23393]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:30 smithi043 ceph-mon[22404]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:30 smithi043 ceph-mon[27243]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:31.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-03T10:21:31.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:31.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-03T10:21:31.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:31.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-03T10:21:31.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:32.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:32 smithi181 ceph-mon[23393]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:32.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:32 smithi181 ceph-mon[23393]: Deploying daemon osd.1 on smithi043 2024-05-03T10:21:32.906 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:32 smithi043 ceph-mon[22404]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:32.906 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:32 smithi043 ceph-mon[22404]: Deploying daemon osd.1 on smithi043 2024-05-03T10:21:32.906 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:32 smithi043 ceph-mon[27243]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:32.906 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:32 smithi043 ceph-mon[27243]: Deploying daemon osd.1 on smithi043 2024-05-03T10:21:34.843 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:34 smithi043 ceph-mon[22404]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:34.843 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:34 smithi043 ceph-mon[27243]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:34.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:34 smithi181 ceph-mon[23393]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:36.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:36 smithi181 ceph-mon[23393]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:36.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:36 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:36.943 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:36 smithi043 ceph-mon[22404]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:36.943 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:36 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:36.943 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:36 smithi043 ceph-mon[27243]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:36.943 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:36 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:37.261 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:21:37 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:21:37] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:21:37.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:37 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:37.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:37 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:37.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:37 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:37.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:37 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:37.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:37 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:37.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:37 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:37.880 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:37.881 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:37.882 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:37 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:38.669 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:38 smithi181 ceph-mon[23393]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:39.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:38 smithi043 ceph-mon[22404]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:39.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:38 smithi043 ceph-mon[27243]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:39.353 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 1 on host 'smithi043' 2024-05-03T10:21:40.008 DEBUG:teuthology.orchestra.run.smithi043:osd.1> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.1.service 2024-05-03T10:21:40.051 INFO:tasks.cephadm:Deploying osd.2 on smithi043 with /dev/vg_nvme/lv_2... 2024-05-03T10:21:40.052 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-03T10:21:40.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:40.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:40.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:40.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:40.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:21:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:21:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: Adjusting osd_memory_target on smithi043 to 7522M 2024-05-03T10:21:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:40 smithi181 ceph-mon[23393]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T10:21:40.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:40.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:40.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:40.627 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: Adjusting osd_memory_target on smithi043 to 7522M 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[22404]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:40.628 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:40.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:21:40.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:21:40.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: Adjusting osd_memory_target on smithi043 to 7522M 2024-05-03T10:21:40.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:40.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:40.629 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:40 smithi043 ceph-mon[27243]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T10:21:41.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:41 smithi043 ceph-mon[22404]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-03T10:21:41.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:41 smithi043 ceph-mon[22404]: osdmap e11: 2 total, 1 up, 2 in 2024-05-03T10:21:41.684 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:41.685 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:41 smithi043 ceph-mon[22404]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:21:41.685 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:41 smithi043 ceph-mon[27243]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-03T10:21:41.685 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:41 smithi043 ceph-mon[27243]: osdmap e11: 2 total, 1 up, 2 in 2024-05-03T10:21:41.685 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:41.685 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:41 smithi043 ceph-mon[27243]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:21:41.685 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:21:41 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1[33294]: 2024-05-03T10:21:41.614+0000 7f28f5068700 -1 osd.1 0 waiting for initial osdmap 2024-05-03T10:21:41.685 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:21:41 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1[33294]: 2024-05-03T10:21:41.617+0000 7f28ef1fe700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T10:21:41.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:41 smithi181 ceph-mon[23393]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-03T10:21:41.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:41 smithi181 ceph-mon[23393]: osdmap e11: 2 total, 1 up, 2 in 2024-05-03T10:21:41.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:41.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:41 smithi181 ceph-mon[23393]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:21:42.160 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-03T10:21:42.160 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-03T10:21:42.160 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-05-03T10:21:42.160 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-05-03T10:21:42.160 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00946202 s, 1.1 GB/s 2024-05-03T10:21:42.160 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-05-03T10:21:42.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[22404]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:42.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[22404]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:21:42.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[22404]: osdmap e12: 2 total, 1 up, 2 in 2024-05-03T10:21:42.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:42.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[27243]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[27243]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:21:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[27243]: osdmap e12: 2 total, 1 up, 2 in 2024-05-03T10:21:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:42.826 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_2 2024-05-03T10:21:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:42 smithi181 ceph-mon[23393]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:42 smithi181 ceph-mon[23393]: from='osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:21:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:42 smithi181 ceph-mon[23393]: osdmap e12: 2 total, 1 up, 2 in 2024-05-03T10:21:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:42 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:42 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:43.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:21:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:21:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:21:43.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[27243]: purged_snaps scrub starts 2024-05-03T10:21:43.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[27243]: purged_snaps scrub ok 2024-05-03T10:21:43.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[27243]: osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390] boot 2024-05-03T10:21:43.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[27243]: osdmap e13: 2 total, 2 up, 2 in 2024-05-03T10:21:43.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:43.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[22404]: purged_snaps scrub starts 2024-05-03T10:21:43.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[22404]: purged_snaps scrub ok 2024-05-03T10:21:43.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[22404]: osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390] boot 2024-05-03T10:21:43.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[22404]: osdmap e13: 2 total, 2 up, 2 in 2024-05-03T10:21:43.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:44.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:43 smithi181 ceph-mon[23393]: purged_snaps scrub starts 2024-05-03T10:21:44.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:43 smithi181 ceph-mon[23393]: purged_snaps scrub ok 2024-05-03T10:21:44.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:43 smithi181 ceph-mon[23393]: osd.1 [v2:172.21.15.43:6810/3187463390,v1:172.21.15.43:6811/3187463390] boot 2024-05-03T10:21:44.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:43 smithi181 ceph-mon[23393]: osdmap e13: 2 total, 2 up, 2 in 2024-05-03T10:21:44.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:43 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:21:44.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:44.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: osdmap e14: 2 total, 2 up, 2 in 2024-05-03T10:21:44.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:44.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: Detected new or changed devices on smithi043 2024-05-03T10:21:44.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: osdmap e14: 2 total, 2 up, 2 in 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: Detected new or changed devices on smithi043 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:44.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:44.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:44.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:44 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:45.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-03T10:21:45.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: osdmap e14: 2 total, 2 up, 2 in 2024-05-03T10:21:45.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:45.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: Detected new or changed devices on smithi043 2024-05-03T10:21:45.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:45.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:45.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:45.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:45.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:45.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:45.683 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:45 smithi043 ceph-mon[22404]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-05-03T10:21:45.683 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:45 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:21:45.683 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:45 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:21:45.683 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:45 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:45.683 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:45 smithi043 ceph-mon[27243]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-05-03T10:21:45.683 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:45 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:21:45.684 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:45 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:21:45.684 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:45 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:46.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:45 smithi181 ceph-mon[23393]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-05-03T10:21:46.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:45 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:21:46.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:45 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:21:46.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:45 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:47.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:46 smithi043 ceph-mon[27243]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:21:47.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:46 smithi043 ceph-mon[27243]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:47.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:46 smithi043 ceph-mon[22404]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:21:47.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:46 smithi043 ceph-mon[22404]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:47.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:46 smithi181 ceph-mon[23393]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:21:47.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:46 smithi181 ceph-mon[23393]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:48.721 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:21:48 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:21:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:21:48.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[22404]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:48.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:48.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:48.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[27243]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:48.998 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:48 smithi181 ceph-mon[23393]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:21:49.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:49.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:21:49.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:50.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:49 smithi043 ceph-mon[22404]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:50.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:49 smithi043 ceph-mon[22404]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-05-03T10:21:50.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:49 smithi043 ceph-mon[22404]: Cluster is now healthy 2024-05-03T10:21:50.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:49 smithi043 ceph-mon[27243]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:50.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:49 smithi043 ceph-mon[27243]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-05-03T10:21:50.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:49 smithi043 ceph-mon[27243]: Cluster is now healthy 2024-05-03T10:21:50.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:49 smithi181 ceph-mon[23393]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:50.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:49 smithi181 ceph-mon[23393]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-05-03T10:21:50.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:49 smithi181 ceph-mon[23393]: Cluster is now healthy 2024-05-03T10:21:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1436935190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05320fef-7e52-4c68-8484-317080e0c4bc"}]: dispatch 2024-05-03T10:21:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1436935190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05320fef-7e52-4c68-8484-317080e0c4bc"}]': finished 2024-05-03T10:21:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[22404]: osdmap e15: 3 total, 2 up, 3 in 2024-05-03T10:21:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:21:51.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/305185058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:21:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1436935190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05320fef-7e52-4c68-8484-317080e0c4bc"}]: dispatch 2024-05-03T10:21:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1436935190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05320fef-7e52-4c68-8484-317080e0c4bc"}]': finished 2024-05-03T10:21:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[27243]: osdmap e15: 3 total, 2 up, 3 in 2024-05-03T10:21:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:21:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:50 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/305185058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:21:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:50 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1436935190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05320fef-7e52-4c68-8484-317080e0c4bc"}]: dispatch 2024-05-03T10:21:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:50 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1436935190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05320fef-7e52-4c68-8484-317080e0c4bc"}]': finished 2024-05-03T10:21:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:50 smithi181 ceph-mon[23393]: osdmap e15: 3 total, 2 up, 3 in 2024-05-03T10:21:51.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:50 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:21:51.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:50 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/305185058' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:21:52.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:51 smithi043 ceph-mon[27243]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:52.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:51 smithi043 ceph-mon[22404]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:52.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:51 smithi181 ceph-mon[23393]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:53.883 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:53 smithi043 ceph-mon[22404]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:53.883 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:53 smithi043 ceph-mon[27243]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:54.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:53 smithi181 ceph-mon[23393]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:55.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:55 smithi043 ceph-mon[22404]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:55.918 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:21:55.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:21:55.919 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:55 smithi043 ceph-mon[27243]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:55.919 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:21:55.919 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:21:56.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:55 smithi181 ceph-mon[23393]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:56.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:21:56.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:21:57.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:56 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-03T10:21:57.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:56 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:57.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:56 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-03T10:21:57.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:56 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:57.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:56 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-03T10:21:57.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:56 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:21:57.484 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:21:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:21:57] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:21:58.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:57 smithi181 ceph-mon[23393]: Deploying daemon osd.2 on smithi043 2024-05-03T10:21:58.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:57 smithi181 ceph-mon[23393]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:58.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:58.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:58.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:57 smithi043 ceph-mon[22404]: Deploying daemon osd.2 on smithi043 2024-05-03T10:21:58.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:57 smithi043 ceph-mon[22404]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:58.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:58.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:58.018 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:57 smithi043 ceph-mon[27243]: Deploying daemon osd.2 on smithi043 2024-05-03T10:21:58.019 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:57 smithi043 ceph-mon[27243]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:58.019 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:58.019 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:21:58.966 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:21:58 smithi181 ceph-mon[23393]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:59.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:21:58 smithi043 ceph-mon[22404]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:21:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:21:58 smithi043 ceph-mon[27243]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:01.788 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[22404]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[27243]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:01.789 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:01 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:01.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:01 smithi181 ceph-mon[23393]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:01.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:01 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:01.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:01 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:01.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:01 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:01.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:01 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:01.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:01 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:03.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:03 smithi181 ceph-mon[23393]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:03.890 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:03 smithi043 ceph-mon[27243]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:03.890 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:03 smithi043 ceph-mon[22404]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:03.922 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 2 on host 'smithi043' 2024-05-03T10:22:04.629 DEBUG:teuthology.orchestra.run.smithi043:osd.2> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.2.service 2024-05-03T10:22:04.632 INFO:tasks.cephadm:Deploying osd.3 on smithi043 with /dev/vg_nvme/lv_1... 2024-05-03T10:22:04.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-03T10:22:05.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:05.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:05.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:05.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:05.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: Adjusting osd_memory_target on smithi043 to 5015M 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='osd.2 [v2:172.21.15.43:6818/260846504,v1:172.21.15.43:6819/260846504]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[27243]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:05.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:05.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:05.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:05.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:05.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: Adjusting osd_memory_target on smithi043 to 5015M 2024-05-03T10:22:05.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:05.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:05.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='osd.2 [v2:172.21.15.43:6818/260846504,v1:172.21.15.43:6819/260846504]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T10:22:05.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T10:22:05.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:04 smithi043 ceph-mon[22404]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:05.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:05.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:05.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:05.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:05.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:05.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:05.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:05.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: Adjusting osd_memory_target on smithi043 to 5015M 2024-05-03T10:22:05.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:05.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:05.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='osd.2 [v2:172.21.15.43:6818/260846504,v1:172.21.15.43:6819/260846504]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T10:22:05.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T10:22:05.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:04 smithi181 ceph-mon[23393]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:06.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[22404]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-03T10:22:06.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[22404]: osdmap e16: 3 total, 2 up, 3 in 2024-05-03T10:22:06.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:06.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[22404]: from='osd.2 [v2:172.21.15.43:6818/260846504,v1:172.21.15.43:6819/260846504]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:06.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[22404]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:06.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[27243]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-03T10:22:06.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[27243]: osdmap e16: 3 total, 2 up, 3 in 2024-05-03T10:22:06.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:06.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[27243]: from='osd.2 [v2:172.21.15.43:6818/260846504,v1:172.21.15.43:6819/260846504]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:06.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:05 smithi043 ceph-mon[27243]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:06.044 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:22:05 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2[36237]: 2024-05-03T10:22:05.943+0000 7fd46b869700 -1 osd.2 0 waiting for initial osdmap 2024-05-03T10:22:06.044 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:22:05 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2[36237]: 2024-05-03T10:22:05.947+0000 7fd4641fc700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T10:22:06.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:05 smithi181 ceph-mon[23393]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-03T10:22:06.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:05 smithi181 ceph-mon[23393]: osdmap e16: 3 total, 2 up, 3 in 2024-05-03T10:22:06.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:05 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:06.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:05 smithi181 ceph-mon[23393]: from='osd.2 [v2:172.21.15.43:6818/260846504,v1:172.21.15.43:6819/260846504]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:06.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:05 smithi181 ceph-mon[23393]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:07.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[22404]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:22:07.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[22404]: osdmap e17: 3 total, 2 up, 3 in 2024-05-03T10:22:07.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:07.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:07.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[22404]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:07.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[27243]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:22:07.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[27243]: osdmap e17: 3 total, 2 up, 3 in 2024-05-03T10:22:07.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:07.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:07.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:06 smithi043 ceph-mon[27243]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:07.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:06 smithi181 ceph-mon[23393]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:22:07.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:06 smithi181 ceph-mon[23393]: osdmap e17: 3 total, 2 up, 3 in 2024-05-03T10:22:07.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:07.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:07.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:06 smithi181 ceph-mon[23393]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-03T10:22:08.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[27243]: purged_snaps scrub starts 2024-05-03T10:22:08.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[27243]: purged_snaps scrub ok 2024-05-03T10:22:08.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:08.077 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[27243]: osd.2 [v2:172.21.15.43:6818/260846504,v1:172.21.15.43:6819/260846504] boot 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[27243]: osdmap e18: 3 total, 3 up, 3 in 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[22404]: purged_snaps scrub starts 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[22404]: purged_snaps scrub ok 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[22404]: osd.2 [v2:172.21.15.43:6818/260846504,v1:172.21.15.43:6819/260846504] boot 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[22404]: osdmap e18: 3 total, 3 up, 3 in 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:08.078 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-03T10:22:08.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:07 smithi181 ceph-mon[23393]: purged_snaps scrub starts 2024-05-03T10:22:08.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:07 smithi181 ceph-mon[23393]: purged_snaps scrub ok 2024-05-03T10:22:08.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:08.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:07 smithi181 ceph-mon[23393]: osd.2 [v2:172.21.15.43:6818/260846504,v1:172.21.15.43:6819/260846504] boot 2024-05-03T10:22:08.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:07 smithi181 ceph-mon[23393]: osdmap e18: 3 total, 3 up, 3 in 2024-05-03T10:22:08.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:22:08.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-03T10:22:08.626 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-03T10:22:08.626 INFO:teuthology.orchestra.run.smithi043.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-03T10:22:08.626 INFO:teuthology.orchestra.run.smithi043.stderr: stderr: 10+0 records in 2024-05-03T10:22:08.626 INFO:teuthology.orchestra.run.smithi043.stderr:10+0 records out 2024-05-03T10:22:08.626 INFO:teuthology.orchestra.run.smithi043.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00962158 s, 1.1 GB/s 2024-05-03T10:22:08.627 INFO:teuthology.orchestra.run.smithi043.stderr:--> Zapping successful for: 2024-05-03T10:22:08.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:22:08 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:22:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:22:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-03T10:22:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: osdmap e19: 3 total, 3 up, 3 in 2024-05-03T10:22:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T10:22:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: Detected new or changed devices on smithi043 2024-05-03T10:22:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:09.266 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: osdmap e19: 3 total, 3 up, 3 in 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:09.267 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: Detected new or changed devices on smithi043 2024-05-03T10:22:09.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:09.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:09.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:09.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:09.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:09.268 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:09.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-03T10:22:09.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: osdmap e19: 3 total, 3 up, 3 in 2024-05-03T10:22:09.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T10:22:09.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:09.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: Detected new or changed devices on smithi043 2024-05-03T10:22:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:09.556 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi043:vg_nvme/lv_1 2024-05-03T10:22:09.792 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:22:09 smithi043 sudo[37392]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T10:22:09.792 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:22:09 smithi043 sudo[37392]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:09.792 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:22:09 smithi043 sudo[37392]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:10.160 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:09 smithi043 ceph-mon[27243]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-05-03T10:22:10.160 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:09 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-03T10:22:10.160 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:09 smithi043 ceph-mon[27243]: osdmap e20: 3 total, 3 up, 3 in 2024-05-03T10:22:10.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:09 smithi043 ceph-mon[22404]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-05-03T10:22:10.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:09 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-03T10:22:10.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:09 smithi043 ceph-mon[22404]: osdmap e20: 3 total, 3 up, 3 in 2024-05-03T10:22:10.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:09 smithi181 ceph-mon[23393]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-05-03T10:22:10.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:09 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-03T10:22:10.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:09 smithi181 ceph-mon[23393]: osdmap e20: 3 total, 3 up, 3 in 2024-05-03T10:22:10.542 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:22:10 smithi043 sudo[37464]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T10:22:10.542 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:22:10 smithi043 sudo[37464]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:10.542 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:22:10 smithi043 sudo[37464]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:10.903 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:22:10 smithi043 sudo[37492]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T10:22:10.903 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:22:10 smithi043 sudo[37492]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:10.903 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:22:10 smithi043 sudo[37492]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:11.179 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:10 smithi043 ceph-mon[22404]: osdmap e21: 3 total, 3 up, 3 in 2024-05-03T10:22:11.179 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:10 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:11.179 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:10 smithi043 ceph-mon[22404]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:11.180 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:10 smithi043 ceph-mon[27243]: osdmap e21: 3 total, 3 up, 3 in 2024-05-03T10:22:11.180 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:10 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:11.180 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:10 smithi043 ceph-mon[27243]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:11.180 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:22:11 smithi043 sudo[37513]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T10:22:11.180 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:22:11 smithi043 sudo[37513]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:11.180 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:22:11 smithi043 sudo[37513]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:11.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:10 smithi181 ceph-mon[23393]: osdmap e21: 3 total, 3 up, 3 in 2024-05-03T10:22:11.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:10 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:11.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:10 smithi181 ceph-mon[23393]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:12.007 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:22:11 smithi043 sudo[37591]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T10:22:12.007 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:22:11 smithi043 sudo[37591]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:12.007 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:22:11 smithi043 sudo[37591]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:12.008 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:22:11 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:22:11] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:22:12.267 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:22:12 smithi043 sudo[37683]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T10:22:12.267 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:22:12 smithi043 sudo[37683]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:12.267 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:22:12 smithi043 sudo[37683]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:12.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 sudo[37703]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T10:22:12.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 sudo[37703]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:12.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 sudo[37703]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:13.260 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 sudo[37709]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 sudo[37709]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 sudo[37709]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:13.261 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:13.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:13.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:13.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-03T10:22:13.262 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[22404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T10:22:13.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:13.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:13.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:22:13.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:22:13.262 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:22:13.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:13.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:13.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:13.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:13.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:13.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:13.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-03T10:22:13.263 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:12 smithi043 ceph-mon[27243]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T10:22:13.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:13.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:13.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi043:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:22:13.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:22:13.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:22:13.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:13.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:13.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:13.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:13.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:13.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:13.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-03T10:22:13.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:12 smithi181 ceph-mon[23393]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T10:22:13.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:13 smithi043 sudo[37737]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T10:22:13.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:13 smithi043 sudo[37737]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:13.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:13 smithi043 sudo[37737]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:14.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:13 smithi043 sudo[37771]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T10:22:14.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:13 smithi043 sudo[37771]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:14.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:13 smithi043 sudo[37771]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:14.341 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 sudo[25880]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T10:22:14.341 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 sudo[25880]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:14.341 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 sudo[25880]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:14.341 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 sudo[25883]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T10:22:14.341 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 sudo[25883]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-03T10:22:14.341 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 sudo[25883]: pam_unix(sudo:session): session closed for user root 2024-05-03T10:22:14.557 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: Cluster is now healthy 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:14.558 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: Cluster is now healthy 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T10:22:14.559 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:22:14.560 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:22:14.560 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:22:14.560 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:14.560 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T10:22:14.560 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:22:14.560 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:22:14.560 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:14 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:22:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T10:22:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-05-03T10:22:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: Cluster is now healthy 2024-05-03T10:22:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:22:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:22:14.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:22:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T10:22:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:22:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:22:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:22:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T10:22:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:22:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:22:14.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:14 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:22:15.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:15 smithi181 ceph-mon[23393]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T10:22:15.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:15 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2457079946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "013775c5-f256-4587-a80f-febbf5992d1f"}]: dispatch 2024-05-03T10:22:15.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:15 smithi181 ceph-mon[23393]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T10:22:15.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:15 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "013775c5-f256-4587-a80f-febbf5992d1f"}]: dispatch 2024-05-03T10:22:15.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:15 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "013775c5-f256-4587-a80f-febbf5992d1f"}]': finished 2024-05-03T10:22:15.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:15 smithi181 ceph-mon[23393]: osdmap e22: 4 total, 3 up, 4 in 2024-05-03T10:22:15.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:15 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:15.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:15 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:15.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:15 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2684331199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:22:15.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[22404]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T10:22:15.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2457079946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "013775c5-f256-4587-a80f-febbf5992d1f"}]: dispatch 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[22404]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "013775c5-f256-4587-a80f-febbf5992d1f"}]: dispatch 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "013775c5-f256-4587-a80f-febbf5992d1f"}]': finished 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[22404]: osdmap e22: 4 total, 3 up, 4 in 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2684331199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[27243]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2457079946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "013775c5-f256-4587-a80f-febbf5992d1f"}]: dispatch 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[27243]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "013775c5-f256-4587-a80f-febbf5992d1f"}]: dispatch 2024-05-03T10:22:15.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "013775c5-f256-4587-a80f-febbf5992d1f"}]': finished 2024-05-03T10:22:15.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[27243]: osdmap e22: 4 total, 3 up, 4 in 2024-05-03T10:22:15.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:15.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:15.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:15 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2684331199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:22:16.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:16 smithi181 ceph-mon[23393]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:16.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:16 smithi043 ceph-mon[22404]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:16.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:16 smithi043 ceph-mon[27243]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:17.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:22:17 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:22:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:22:18.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:18.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:18.553 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:18 smithi181 ceph-mon[23393]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:18.578 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:18.578 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:18.578 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:18 smithi043 ceph-mon[27243]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:18.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:18.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:18.578 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:18 smithi043 ceph-mon[22404]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:20.966 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:20 smithi043 ceph-mon[27243]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-05-03T10:22:20.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:20 smithi043 ceph-mon[22404]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-05-03T10:22:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:20 smithi181 ceph-mon[23393]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-05-03T10:22:22.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:21 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-03T10:22:22.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:21 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:22.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:21 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-03T10:22:22.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:21 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:22.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:21 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-03T10:22:22.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:21 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:22.943 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:22 smithi043 ceph-mon[22404]: Deploying daemon osd.3 on smithi043 2024-05-03T10:22:22.943 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:22 smithi043 ceph-mon[22404]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-05-03T10:22:22.943 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:22 smithi043 ceph-mon[27243]: Deploying daemon osd.3 on smithi043 2024-05-03T10:22:22.943 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:22 smithi043 ceph-mon[27243]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-05-03T10:22:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:22 smithi181 ceph-mon[23393]: Deploying daemon osd.3 on smithi043 2024-05-03T10:22:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:22 smithi181 ceph-mon[23393]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-05-03T10:22:25.023 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:24 smithi043 ceph-mon[22404]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-03T10:22:25.023 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:24 smithi043 ceph-mon[27243]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-03T10:22:25.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:24 smithi181 ceph-mon[23393]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-03T10:22:26.957 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:26 smithi043 ceph-mon[22404]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 69 KiB/s wr, 4 op/s 2024-05-03T10:22:26.957 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:26 smithi043 ceph-mon[27243]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 69 KiB/s wr, 4 op/s 2024-05-03T10:22:27.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:26 smithi181 ceph-mon[23393]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 69 KiB/s wr, 4 op/s 2024-05-03T10:22:28.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:28.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:28.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:28.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:28.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:28.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[27243]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-03T10:22:28.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:28.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:28.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:28.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:28.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:28.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:28 smithi043 ceph-mon[22404]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-03T10:22:28.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:28 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:28.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:28 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:28.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:28 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:28.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:28 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:28.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:28 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:28.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:28 smithi181 ceph-mon[23393]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-03T10:22:30.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:30 smithi043 ceph-mon[22404]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-03T10:22:30.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:30 smithi043 ceph-mon[22404]: from='osd.3 [v2:172.21.15.43:6826/1619280720,v1:172.21.15.43:6827/1619280720]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T10:22:30.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:30 smithi043 ceph-mon[22404]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T10:22:30.917 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:30 smithi043 ceph-mon[27243]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-03T10:22:30.917 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:30 smithi043 ceph-mon[27243]: from='osd.3 [v2:172.21.15.43:6826/1619280720,v1:172.21.15.43:6827/1619280720]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T10:22:30.917 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:30 smithi043 ceph-mon[27243]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T10:22:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:30 smithi181 ceph-mon[23393]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-03T10:22:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:30 smithi181 ceph-mon[23393]: from='osd.3 [v2:172.21.15.43:6826/1619280720,v1:172.21.15.43:6827/1619280720]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T10:22:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:30 smithi181 ceph-mon[23393]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T10:22:31.214 INFO:teuthology.orchestra.run.smithi043.stdout:Created osd(s) 3 on host 'smithi043' 2024-05-03T10:22:31.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:22:31 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:22:31] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:22:31.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-03T10:22:31.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: osdmap e23: 4 total, 3 up, 4 in 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='osd.3 [v2:172.21.15.43:6826/1619280720,v1:172.21.15.43:6827/1619280720]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:31.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: osdmap e23: 4 total, 3 up, 4 in 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='osd.3 [v2:172.21.15.43:6826/1619280720,v1:172.21.15.43:6827/1619280720]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:31.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:31.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:31.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:31.795 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:31 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:31.886 DEBUG:teuthology.orchestra.run.smithi043:osd.3> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.3.service 2024-05-03T10:22:31.889 INFO:tasks.cephadm:Deploying osd.4 on smithi181 with /dev/vg_nvme/lv_4... 2024-05-03T10:22:31.889 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-03T10:22:32.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-03T10:22:32.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: osdmap e23: 4 total, 3 up, 4 in 2024-05-03T10:22:32.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:32.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='osd.3 [v2:172.21.15.43:6826/1619280720,v1:172.21.15.43:6827/1619280720]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:32.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]: dispatch 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:32.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:31 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:32.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:22:32 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:22:32] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:22:32.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: Adjusting osd_memory_target on smithi043 to 3761M 2024-05-03T10:22:32.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: Detected new or changed devices on smithi043 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: osdmap e24: 4 total, 3 up, 4 in 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:32.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:32.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:32.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:32.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:32.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:32 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: Adjusting osd_memory_target on smithi043 to 3761M 2024-05-03T10:22:33.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:33.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: Detected new or changed devices on smithi043 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: osdmap e24: 4 total, 3 up, 4 in 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:33.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: Adjusting osd_memory_target on smithi043 to 3761M 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: Detected new or changed devices on smithi043 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi043", "root=default"]}]': finished 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: osdmap e24: 4 total, 3 up, 4 in 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:33.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:33.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:33.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:33.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:33.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:33.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:32 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:33.854 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:33 smithi181 ceph-mon[23393]: purged_snaps scrub starts 2024-05-03T10:22:33.854 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:33 smithi181 ceph-mon[23393]: purged_snaps scrub ok 2024-05-03T10:22:33.854 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:33 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:33.854 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:33 smithi181 ceph-mon[23393]: osd.3 [v2:172.21.15.43:6826/1619280720,v1:172.21.15.43:6827/1619280720] boot 2024-05-03T10:22:33.854 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:33 smithi181 ceph-mon[23393]: osdmap e25: 4 total, 4 up, 4 in 2024-05-03T10:22:33.854 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:33 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:33.855 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-03T10:22:33.855 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-03T10:22:33.855 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-05-03T10:22:33.855 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-05-03T10:22:33.855 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119923 s, 874 MB/s 2024-05-03T10:22:33.855 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-05-03T10:22:34.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[22404]: purged_snaps scrub starts 2024-05-03T10:22:34.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[22404]: purged_snaps scrub ok 2024-05-03T10:22:34.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:34.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[22404]: osd.3 [v2:172.21.15.43:6826/1619280720,v1:172.21.15.43:6827/1619280720] boot 2024-05-03T10:22:34.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[22404]: osdmap e25: 4 total, 4 up, 4 in 2024-05-03T10:22:34.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:34.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[27243]: purged_snaps scrub starts 2024-05-03T10:22:34.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[27243]: purged_snaps scrub ok 2024-05-03T10:22:34.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:34.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[27243]: osd.3 [v2:172.21.15.43:6826/1619280720,v1:172.21.15.43:6827/1619280720] boot 2024-05-03T10:22:34.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[27243]: osdmap e25: 4 total, 4 up, 4 in 2024-05-03T10:22:34.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:33 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:22:34.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2024-05-03T10:22:35.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:34 smithi043 ceph-mon[22404]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:35.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:34 smithi043 ceph-mon[22404]: osdmap e26: 4 total, 4 up, 4 in 2024-05-03T10:22:35.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:34 smithi043 ceph-mon[27243]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:35.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:34 smithi043 ceph-mon[27243]: osdmap e26: 4 total, 4 up, 4 in 2024-05-03T10:22:35.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:34 smithi181 ceph-mon[23393]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-03T10:22:35.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:34 smithi181 ceph-mon[23393]: osdmap e26: 4 total, 4 up, 4 in 2024-05-03T10:22:36.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:36 smithi181 ceph-mon[23393]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:36.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:36 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:22:36.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:36 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:22:36.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:36 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:37.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:36 smithi043 ceph-mon[22404]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:37.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:36 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:22:37.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:36 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:22:37.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:36 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:37.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:36 smithi043 ceph-mon[27243]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:37.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:36 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:22:37.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:36 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:22:37.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:36 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:37.725 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:22:37 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:22:37] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:22:37.927 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:37 smithi181 ceph-mon[23393]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:22:37.927 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:37 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:37.927 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:37 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:38.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:37 smithi043 ceph-mon[22404]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:22:38.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:37 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:38.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:37 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:38.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:37 smithi043 ceph-mon[27243]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:22:38.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:37 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:38.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:37 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:38.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:38 smithi181 ceph-mon[23393]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:39.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:38 smithi043 ceph-mon[22404]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:39.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:38 smithi043 ceph-mon[27243]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:40.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/1941330407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60188084-c744-4a5a-8593-2fe14c1a3349"}]: dispatch 2024-05-03T10:22:40.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60188084-c744-4a5a-8593-2fe14c1a3349"}]: dispatch 2024-05-03T10:22:40.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60188084-c744-4a5a-8593-2fe14c1a3349"}]': finished 2024-05-03T10:22:40.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[22404]: osdmap e27: 5 total, 4 up, 5 in 2024-05-03T10:22:40.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:40.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/1941330407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60188084-c744-4a5a-8593-2fe14c1a3349"}]: dispatch 2024-05-03T10:22:40.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60188084-c744-4a5a-8593-2fe14c1a3349"}]: dispatch 2024-05-03T10:22:40.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60188084-c744-4a5a-8593-2fe14c1a3349"}]': finished 2024-05-03T10:22:40.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[27243]: osdmap e27: 5 total, 4 up, 5 in 2024-05-03T10:22:40.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:39 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:40.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:39 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/1941330407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60188084-c744-4a5a-8593-2fe14c1a3349"}]: dispatch 2024-05-03T10:22:40.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:39 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60188084-c744-4a5a-8593-2fe14c1a3349"}]: dispatch 2024-05-03T10:22:40.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:39 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60188084-c744-4a5a-8593-2fe14c1a3349"}]': finished 2024-05-03T10:22:40.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:39 smithi181 ceph-mon[23393]: osdmap e27: 5 total, 4 up, 5 in 2024-05-03T10:22:40.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:39 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:41.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:40 smithi043 ceph-mon[22404]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:41.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:40 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/2797723349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:22:41.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:40 smithi043 ceph-mon[27243]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:41.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:40 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/2797723349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:22:41.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:40 smithi181 ceph-mon[23393]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:41.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:40 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/2797723349' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:22:42.852 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:42 smithi181 ceph-mon[23393]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:43.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:42 smithi043 ceph-mon[22404]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:43.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:42 smithi043 ceph-mon[27243]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:44.753 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:44 smithi181 ceph-mon[23393]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:44.955 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:44 smithi043 ceph-mon[22404]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:44.955 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:44 smithi043 ceph-mon[27243]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:46.965 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:46 smithi043 ceph-mon[22404]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:46.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-03T10:22:46.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:46.966 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:46 smithi043 ceph-mon[27243]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:46.966 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-03T10:22:46.966 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:46.987 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:46 smithi181 ceph-mon[23393]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:46.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-03T10:22:46.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:47.705 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:22:47 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:22:47] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:22:47.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:47 smithi181 ceph-mon[23393]: Deploying daemon osd.4 on smithi181 2024-05-03T10:22:47.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:47 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:47.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:47 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:48.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:47 smithi043 ceph-mon[27243]: Deploying daemon osd.4 on smithi181 2024-05-03T10:22:48.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:47 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:48.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:47 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:48.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:47 smithi043 ceph-mon[22404]: Deploying daemon osd.4 on smithi181 2024-05-03T10:22:48.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:47 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:48.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:47 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:48.965 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:48 smithi181 ceph-mon[23393]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:49.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:48 smithi043 ceph-mon[22404]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:49.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:48 smithi043 ceph-mon[27243]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:50.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:50 smithi181 ceph-mon[23393]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:50 smithi043 ceph-mon[22404]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:51.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:50 smithi043 ceph-mon[27243]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:51.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:51.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:51.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:51.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:51.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:52.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:52.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:52.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:52.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:52.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:52 smithi181 ceph-mon[23393]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:53.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:52 smithi043 ceph-mon[22404]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:53.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:52 smithi043 ceph-mon[27243]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:54.033 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 4 on host 'smithi181' 2024-05-03T10:22:54.614 DEBUG:teuthology.orchestra.run.smithi181:osd.4> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.4.service 2024-05-03T10:22:54.617 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_3... 2024-05-03T10:22:54.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-03T10:22:54.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:54.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:54.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:54.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:54.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:54.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:54.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:54.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:54.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: from='osd.4 [v2:172.21.15.181:6800/3176358771,v1:172.21.15.181:6801/3176358771]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T10:22:54.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:54 smithi181 ceph-mon[23393]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T10:22:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: from='osd.4 [v2:172.21.15.181:6800/3176358771,v1:172.21.15.181:6801/3176358771]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[22404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: from='osd.4 [v2:172.21.15.181:6800/3176358771,v1:172.21.15.181:6801/3176358771]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T10:22:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:54 smithi043 ceph-mon[27243]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T10:22:55.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[27243]: Adjusting osd_memory_target on smithi181 to 16064M 2024-05-03T10:22:55.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[27243]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-03T10:22:55.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[27243]: osdmap e28: 5 total, 4 up, 5 in 2024-05-03T10:22:55.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:55.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[27243]: from='osd.4 [v2:172.21.15.181:6800/3176358771,v1:172.21.15.181:6801/3176358771]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:22:55.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[27243]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:22:55.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:22:55.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:22:55.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[22404]: Adjusting osd_memory_target on smithi181 to 16064M 2024-05-03T10:22:55.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[22404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-03T10:22:55.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[22404]: osdmap e28: 5 total, 4 up, 5 in 2024-05-03T10:22:55.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:55.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[22404]: from='osd.4 [v2:172.21.15.181:6800/3176358771,v1:172.21.15.181:6801/3176358771]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:22:55.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[22404]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:22:55.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:22:55.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:22:55.897 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:55 smithi181 ceph-mon[23393]: Adjusting osd_memory_target on smithi181 to 16064M 2024-05-03T10:22:55.897 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:55 smithi181 ceph-mon[23393]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-03T10:22:55.897 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:55 smithi181 ceph-mon[23393]: osdmap e28: 5 total, 4 up, 5 in 2024-05-03T10:22:55.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:55.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:55 smithi181 ceph-mon[23393]: from='osd.4 [v2:172.21.15.181:6800/3176358771,v1:172.21.15.181:6801/3176358771]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:22:55.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:55 smithi181 ceph-mon[23393]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:22:55.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:22:55.898 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:22:56.174 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:22:56 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:22:56.073+0000 7fd22bc49700 -1 osd.4 0 waiting for initial osdmap 2024-05-03T10:22:56.174 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:22:56 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:22:56.080+0000 7fd2285e4700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T10:22:56.685 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-03T10:22:56.685 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-03T10:22:56.685 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-05-03T10:22:56.685 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-05-03T10:22:56.685 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0174565 s, 601 MB/s 2024-05-03T10:22:56.686 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-05-03T10:22:56.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:56 smithi181 ceph-mon[23393]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:56.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:56 smithi181 ceph-mon[23393]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:22:56.979 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:56 smithi181 ceph-mon[23393]: osdmap e29: 5 total, 4 up, 5 in 2024-05-03T10:22:56.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:56 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:56.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:56 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:56.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:56 smithi181 ceph-mon[23393]: osd.4 [v2:172.21.15.181:6800/3176358771,v1:172.21.15.181:6801/3176358771] boot 2024-05-03T10:22:56.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:56 smithi181 ceph-mon[23393]: osdmap e30: 5 total, 5 up, 5 in 2024-05-03T10:22:56.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:56 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:57.013 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[27243]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:57.013 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[27243]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[27243]: osdmap e29: 5 total, 4 up, 5 in 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[27243]: osd.4 [v2:172.21.15.181:6800/3176358771,v1:172.21.15.181:6801/3176358771] boot 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[27243]: osdmap e30: 5 total, 5 up, 5 in 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[22404]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[22404]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[22404]: osdmap e29: 5 total, 4 up, 5 in 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:57.014 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[22404]: osd.4 [v2:172.21.15.181:6800/3176358771,v1:172.21.15.181:6801/3176358771] boot 2024-05-03T10:22:57.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[22404]: osdmap e30: 5 total, 5 up, 5 in 2024-05-03T10:22:57.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:56 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:22:57.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:22:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:22:57] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:22:57.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[27243]: purged_snaps scrub starts 2024-05-03T10:22:57.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[27243]: purged_snaps scrub ok 2024-05-03T10:22:57.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[27243]: osdmap e31: 5 total, 5 up, 5 in 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[22404]: purged_snaps scrub starts 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[22404]: purged_snaps scrub ok 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:57.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:57.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:57.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:57.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:57 smithi043 ceph-mon[22404]: osdmap e31: 5 total, 5 up, 5 in 2024-05-03T10:22:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:57 smithi181 ceph-mon[23393]: purged_snaps scrub starts 2024-05-03T10:22:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:57 smithi181 ceph-mon[23393]: purged_snaps scrub ok 2024-05-03T10:22:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:57.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:57.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:57.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:57.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:57.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:57.825 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:57 smithi181 ceph-mon[23393]: osdmap e31: 5 total, 5 up, 5 in 2024-05-03T10:22:57.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2024-05-03T10:22:58.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:22:58 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:22:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:22:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:58 smithi181 ceph-mon[23393]: Detected new or changed devices on smithi181 2024-05-03T10:22:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:58 smithi181 ceph-mon[23393]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:22:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:58 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:58.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:58 smithi181 ceph-mon[23393]: osdmap e32: 5 total, 5 up, 5 in 2024-05-03T10:22:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[27243]: Detected new or changed devices on smithi181 2024-05-03T10:22:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[27243]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:22:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[27243]: osdmap e32: 5 total, 5 up, 5 in 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[22404]: Detected new or changed devices on smithi181 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[22404]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:22:59.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:58 smithi043 ceph-mon[22404]: osdmap e32: 5 total, 5 up, 5 in 2024-05-03T10:23:00.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:59 smithi043 ceph-mon[27243]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:00.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:22:59 smithi043 ceph-mon[27243]: osdmap e33: 5 total, 5 up, 5 in 2024-05-03T10:23:00.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:59 smithi043 ceph-mon[22404]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:00.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:22:59 smithi043 ceph-mon[22404]: osdmap e33: 5 total, 5 up, 5 in 2024-05-03T10:23:00.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:59 smithi181 ceph-mon[23393]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:00.079 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:22:59 smithi181 ceph-mon[23393]: osdmap e33: 5 total, 5 up, 5 in 2024-05-03T10:23:01.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:00 smithi181 ceph-mon[23393]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:23:01.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:00 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:23:01.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:00 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:23:01.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:00 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:01.040 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:00 smithi181 ceph-mon[23393]: osdmap e34: 5 total, 5 up, 5 in 2024-05-03T10:23:01.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[27243]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:23:01.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:23:01.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:23:01.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:01.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[27243]: osdmap e34: 5 total, 5 up, 5 in 2024-05-03T10:23:01.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[22404]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:23:01.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:23:01.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:23:01.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:01.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:00 smithi043 ceph-mon[22404]: osdmap e34: 5 total, 5 up, 5 in 2024-05-03T10:23:01.981 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:01 smithi181 ceph-mon[23393]: pgmap v119: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 1/6 objects misplaced (16.667%); 56 KiB/s, 0 objects/s recovering 2024-05-03T10:23:02.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:01 smithi043 ceph-mon[22404]: pgmap v119: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 1/6 objects misplaced (16.667%); 56 KiB/s, 0 objects/s recovering 2024-05-03T10:23:02.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:01 smithi043 ceph-mon[27243]: pgmap v119: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 1/6 objects misplaced (16.667%); 56 KiB/s, 0 objects/s recovering 2024-05-03T10:23:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/1430939253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6e7152a-0120-43d4-b099-38a03f9947b9"}]: dispatch 2024-05-03T10:23:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6e7152a-0120-43d4-b099-38a03f9947b9"}]: dispatch 2024-05-03T10:23:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6e7152a-0120-43d4-b099-38a03f9947b9"}]': finished 2024-05-03T10:23:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[22404]: osdmap e35: 6 total, 5 up, 6 in 2024-05-03T10:23:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/1430939253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6e7152a-0120-43d4-b099-38a03f9947b9"}]: dispatch 2024-05-03T10:23:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6e7152a-0120-43d4-b099-38a03f9947b9"}]: dispatch 2024-05-03T10:23:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6e7152a-0120-43d4-b099-38a03f9947b9"}]': finished 2024-05-03T10:23:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[27243]: osdmap e35: 6 total, 5 up, 6 in 2024-05-03T10:23:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:02 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:02 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/1430939253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6e7152a-0120-43d4-b099-38a03f9947b9"}]: dispatch 2024-05-03T10:23:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:02 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b6e7152a-0120-43d4-b099-38a03f9947b9"}]: dispatch 2024-05-03T10:23:03.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:02 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b6e7152a-0120-43d4-b099-38a03f9947b9"}]': finished 2024-05-03T10:23:03.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:02 smithi181 ceph-mon[23393]: osdmap e35: 6 total, 5 up, 6 in 2024-05-03T10:23:03.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:02 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:04.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:03 smithi043 ceph-mon[22404]: pgmap v122: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 1/6 objects misplaced (16.667%); 49 KiB/s, 0 objects/s recovering 2024-05-03T10:23:04.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:03 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/4165365354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:23:04.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:03 smithi043 ceph-mon[27243]: pgmap v122: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 1/6 objects misplaced (16.667%); 49 KiB/s, 0 objects/s recovering 2024-05-03T10:23:04.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:03 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/4165365354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:23:04.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:03 smithi181 ceph-mon[23393]: pgmap v122: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 1/6 objects misplaced (16.667%); 49 KiB/s, 0 objects/s recovering 2024-05-03T10:23:04.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:03 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/4165365354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:23:05.988 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:05 smithi181 ceph-mon[23393]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-05-03T10:23:05.997 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:05 smithi043 ceph-mon[22404]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-05-03T10:23:05.997 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:05 smithi043 ceph-mon[27243]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-05-03T10:23:07.734 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:23:07 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:23:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:23:08.003 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:07 smithi181 ceph-mon[23393]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-03T10:23:08.003 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:08.003 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:08.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:07 smithi043 ceph-mon[22404]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-03T10:23:08.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:08.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:08.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:07 smithi043 ceph-mon[27243]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-03T10:23:08.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:08.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:10.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:09 smithi181 ceph-mon[23393]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-05-03T10:23:10.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:09 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-03T10:23:10.038 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:09 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:10.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:09 smithi043 ceph-mon[22404]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-05-03T10:23:10.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:09 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-03T10:23:10.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:09 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:10.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:09 smithi043 ceph-mon[27243]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-05-03T10:23:10.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:09 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-03T10:23:10.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:09 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:10 smithi043 ceph-mon[22404]: Deploying daemon osd.5 on smithi181 2024-05-03T10:23:11.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:10 smithi043 ceph-mon[27243]: Deploying daemon osd.5 on smithi181 2024-05-03T10:23:11.060 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:10 smithi181 ceph-mon[23393]: Deploying daemon osd.5 on smithi181 2024-05-03T10:23:12.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:11 smithi043 ceph-mon[22404]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 34 KiB/s, 0 objects/s recovering 2024-05-03T10:23:12.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:11 smithi043 ceph-mon[27243]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 34 KiB/s, 0 objects/s recovering 2024-05-03T10:23:12.096 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:11 smithi181 ceph-mon[23393]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 34 KiB/s, 0 objects/s recovering 2024-05-03T10:23:14.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:13 smithi181 ceph-mon[23393]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-05-03T10:23:14.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:13 smithi043 ceph-mon[22404]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-05-03T10:23:14.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:13 smithi043 ceph-mon[27243]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-05-03T10:23:15.522 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:15 smithi181 ceph-mon[23393]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 27 KiB/s, 0 objects/s recovering 2024-05-03T10:23:15.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:15 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:15.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:15 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:15.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:15 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:15.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:15 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:15.523 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:15 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:15.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[27243]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 27 KiB/s, 0 objects/s recovering 2024-05-03T10:23:15.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:15.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:15.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:15.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:15.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[22404]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 27 KiB/s, 0 objects/s recovering 2024-05-03T10:23:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:15.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:15 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:16.955 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2024-05-03T10:23:17.607 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.5.service 2024-05-03T10:23:17.610 INFO:tasks.cephadm:Deploying osd.6 on smithi181 with /dev/vg_nvme/lv_2... 2024-05-03T10:23:17.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-03T10:23:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:17.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:17.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:17.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='osd.5 [v2:172.21.15.181:6808/2900949392,v1:172.21.15.181:6809/2900949392]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T10:23:17.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:17 smithi181 ceph-mon[23393]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T10:23:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='osd.5 [v2:172.21.15.181:6808/2900949392,v1:172.21.15.181:6809/2900949392]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[22404]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:18.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:18.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:18.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='osd.5 [v2:172.21.15.181:6808/2900949392,v1:172.21.15.181:6809/2900949392]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T10:23:18.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:17 smithi043 ceph-mon[27243]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T10:23:18.502 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:23:18 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:23:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:23:18.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: Adjusting osd_memory_target on smithi181 to 8032M 2024-05-03T10:23:18.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-03T10:23:18.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: osdmap e36: 6 total, 5 up, 6 in 2024-05-03T10:23:18.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='osd.5 [v2:172.21.15.181:6808/2900949392,v1:172.21.15.181:6809/2900949392]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: Adjusting osd_memory_target on smithi181 to 8032M 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: osdmap e36: 6 total, 5 up, 6 in 2024-05-03T10:23:18.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:18.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='osd.5 [v2:172.21.15.181:6808/2900949392,v1:172.21.15.181:6809/2900949392]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:18.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:18.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:18.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:18.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:18.794 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:18 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: Adjusting osd_memory_target on smithi181 to 8032M 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: osdmap e36: 6 total, 5 up, 6 in 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='osd.5 [v2:172.21.15.181:6808/2900949392,v1:172.21.15.181:6809/2900949392]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:18.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:18 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:19.185 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:23:18 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:23:18.986+0000 7f743ce9a700 -1 osd.5 0 waiting for initial osdmap 2024-05-03T10:23:19.185 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:23:18 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:23:18.992+0000 7f7437831700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T10:23:19.680 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-03T10:23:19.680 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-03T10:23:19.681 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-05-03T10:23:19.681 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-05-03T10:23:19.681 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.014132 s, 742 MB/s 2024-05-03T10:23:19.681 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-05-03T10:23:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:19 smithi181 ceph-mon[23393]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:19 smithi181 ceph-mon[23393]: Detected new or changed devices on smithi181 2024-05-03T10:23:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:19 smithi181 ceph-mon[23393]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:23:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:19 smithi181 ceph-mon[23393]: osdmap e37: 6 total, 5 up, 6 in 2024-05-03T10:23:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:20.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[22404]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:20.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[22404]: Detected new or changed devices on smithi181 2024-05-03T10:23:20.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[22404]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:23:20.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[22404]: osdmap e37: 6 total, 5 up, 6 in 2024-05-03T10:23:20.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:20.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:20.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[27243]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:20.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[27243]: Detected new or changed devices on smithi181 2024-05-03T10:23:20.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[27243]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:23:20.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[27243]: osdmap e37: 6 total, 5 up, 6 in 2024-05-03T10:23:20.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:20.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:20.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2024-05-03T10:23:21.115 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:20 smithi181 ceph-mon[23393]: purged_snaps scrub starts 2024-05-03T10:23:21.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:20 smithi181 ceph-mon[23393]: purged_snaps scrub ok 2024-05-03T10:23:21.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:20 smithi181 ceph-mon[23393]: osd.5 [v2:172.21.15.181:6808/2900949392,v1:172.21.15.181:6809/2900949392] boot 2024-05-03T10:23:21.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:20 smithi181 ceph-mon[23393]: osdmap e38: 6 total, 6 up, 6 in 2024-05-03T10:23:21.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:20 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:21.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:20 smithi181 ceph-mon[23393]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:21.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[22404]: purged_snaps scrub starts 2024-05-03T10:23:21.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[22404]: purged_snaps scrub ok 2024-05-03T10:23:21.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[22404]: osd.5 [v2:172.21.15.181:6808/2900949392,v1:172.21.15.181:6809/2900949392] boot 2024-05-03T10:23:21.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[22404]: osdmap e38: 6 total, 6 up, 6 in 2024-05-03T10:23:21.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:21.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[22404]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:21.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[27243]: purged_snaps scrub starts 2024-05-03T10:23:21.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[27243]: purged_snaps scrub ok 2024-05-03T10:23:21.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[27243]: osd.5 [v2:172.21.15.181:6808/2900949392,v1:172.21.15.181:6809/2900949392] boot 2024-05-03T10:23:21.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[27243]: osdmap e38: 6 total, 6 up, 6 in 2024-05-03T10:23:21.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:23:21.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:20 smithi043 ceph-mon[27243]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-03T10:23:22.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:21 smithi043 ceph-mon[22404]: osdmap e39: 6 total, 6 up, 6 in 2024-05-03T10:23:22.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:21 smithi043 ceph-mon[27243]: osdmap e39: 6 total, 6 up, 6 in 2024-05-03T10:23:22.104 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:22 smithi181 ceph-mon[23393]: osdmap e39: 6 total, 6 up, 6 in 2024-05-03T10:23:22.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:23:22 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:23:22] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:23:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: osdmap e40: 6 total, 6 up, 6 in 2024-05-03T10:23:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:23:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:23:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:23:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:23 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:23.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: osdmap e40: 6 total, 6 up, 6 in 2024-05-03T10:23:23.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:23.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:23.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:23.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:23.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:23.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:23.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:23.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:23:23.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:23:23.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:23:23.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:23.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: osdmap e40: 6 total, 6 up, 6 in 2024-05-03T10:23:23.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:23.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:23.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:23.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:23.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:23.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:23.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:23.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:23:23.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:23:23.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:23:23.295 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:23 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:24.197 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:24 smithi181 ceph-mon[23393]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-03T10:23:24.197 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:24 smithi181 ceph-mon[23393]: osdmap e41: 6 total, 6 up, 6 in 2024-05-03T10:23:24.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:24 smithi043 ceph-mon[22404]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-03T10:23:24.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:24 smithi043 ceph-mon[22404]: osdmap e41: 6 total, 6 up, 6 in 2024-05-03T10:23:24.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:24 smithi043 ceph-mon[27243]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-03T10:23:24.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:24 smithi043 ceph-mon[27243]: osdmap e41: 6 total, 6 up, 6 in 2024-05-03T10:23:25.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:25 smithi181 ceph-mon[23393]: osdmap e42: 6 total, 6 up, 6 in 2024-05-03T10:23:25.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:25 smithi181 ceph-mon[23393]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:25.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:25 smithi043 ceph-mon[27243]: osdmap e42: 6 total, 6 up, 6 in 2024-05-03T10:23:25.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:25 smithi043 ceph-mon[27243]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:25.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:25 smithi043 ceph-mon[22404]: osdmap e42: 6 total, 6 up, 6 in 2024-05-03T10:23:25.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:25 smithi043 ceph-mon[22404]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:26.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:26 smithi181 ceph-mon[23393]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-03T10:23:26.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:26 smithi181 ceph-mon[23393]: Cluster is now healthy 2024-05-03T10:23:26.523 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:26 smithi043 ceph-mon[22404]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-03T10:23:26.523 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:26 smithi043 ceph-mon[22404]: Cluster is now healthy 2024-05-03T10:23:26.524 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:26 smithi043 ceph-mon[27243]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-03T10:23:26.524 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:26 smithi043 ceph-mon[27243]: Cluster is now healthy 2024-05-03T10:23:27.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:27 smithi181 ceph-mon[23393]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-03T10:23:27.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:27 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/3207069030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85eb0fcf-b686-4e6c-a116-9dc7b55379ac"}]: dispatch 2024-05-03T10:23:27.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:27 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85eb0fcf-b686-4e6c-a116-9dc7b55379ac"}]: dispatch 2024-05-03T10:23:27.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:27 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85eb0fcf-b686-4e6c-a116-9dc7b55379ac"}]': finished 2024-05-03T10:23:27.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:27 smithi181 ceph-mon[23393]: osdmap e43: 7 total, 6 up, 7 in 2024-05-03T10:23:27.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:27 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:27.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:27 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/335303717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:23:28.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:23:27 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:23:27] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:23:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[22404]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-03T10:23:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/3207069030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85eb0fcf-b686-4e6c-a116-9dc7b55379ac"}]: dispatch 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85eb0fcf-b686-4e6c-a116-9dc7b55379ac"}]: dispatch 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85eb0fcf-b686-4e6c-a116-9dc7b55379ac"}]': finished 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[22404]: osdmap e43: 7 total, 6 up, 7 in 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/335303717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[27243]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/3207069030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85eb0fcf-b686-4e6c-a116-9dc7b55379ac"}]: dispatch 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85eb0fcf-b686-4e6c-a116-9dc7b55379ac"}]: dispatch 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85eb0fcf-b686-4e6c-a116-9dc7b55379ac"}]': finished 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[27243]: osdmap e43: 7 total, 6 up, 7 in 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:27 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/335303717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:23:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:28 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:28 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:28 smithi043 ceph-mon[22404]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T10:23:29.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:28 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:29.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:28 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:29.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:28 smithi043 ceph-mon[27243]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T10:23:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:28 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:28 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:28 smithi181 ceph-mon[23393]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T10:23:31.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:31 smithi181 ceph-mon[23393]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-05-03T10:23:31.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:31 smithi043 ceph-mon[22404]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-05-03T10:23:31.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:31 smithi043 ceph-mon[27243]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-05-03T10:23:33.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:33 smithi181 ceph-mon[23393]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-03T10:23:33.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:33 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-03T10:23:33.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:33 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:34.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:33 smithi043 ceph-mon[22404]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-03T10:23:34.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:33 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-03T10:23:34.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:33 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:34.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:33 smithi043 ceph-mon[27243]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-03T10:23:34.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:33 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-03T10:23:34.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:33 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:34.672 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:34 smithi181 ceph-mon[23393]: Deploying daemon osd.6 on smithi181 2024-05-03T10:23:35.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:34 smithi043 ceph-mon[22404]: Deploying daemon osd.6 on smithi181 2024-05-03T10:23:35.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:34 smithi043 ceph-mon[27243]: Deploying daemon osd.6 on smithi181 2024-05-03T10:23:35.626 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:35 smithi181 ceph-mon[23393]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-03T10:23:36.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:35 smithi043 ceph-mon[22404]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-03T10:23:36.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:35 smithi043 ceph-mon[27243]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-03T10:23:37.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:37 smithi181 ceph-mon[23393]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:38.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:37 smithi043 ceph-mon[22404]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:38.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:37 smithi043 ceph-mon[27243]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:39.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:39.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:39.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:39.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:39.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:39.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[22404]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:39.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:39.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:39.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:39.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:39.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:39.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:38 smithi043 ceph-mon[27243]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:39.311 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:38 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:39.311 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:38 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:39.311 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:38 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:39.311 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:38 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:39.311 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:38 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:39.311 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:38 smithi181 ceph-mon[23393]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:40.769 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 6 on host 'smithi181' 2024-05-03T10:23:41.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='osd.6 [v2:172.21.15.181:6816/4115880036,v1:172.21.15.181:6817/4115880036]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T10:23:41.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:41 smithi181 ceph-mon[23393]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T10:23:41.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:41.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:41.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='osd.6 [v2:172.21.15.181:6816/4115880036,v1:172.21.15.181:6817/4115880036]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[27243]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:41.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:41.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:41.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:41.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:41.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:41.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:23:41.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:41.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:41.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='osd.6 [v2:172.21.15.181:6816/4115880036,v1:172.21.15.181:6817/4115880036]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T10:23:41.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:41 smithi043 ceph-mon[22404]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T10:23:42.722 DEBUG:teuthology.orchestra.run.smithi181:osd.6> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.6.service 2024-05-03T10:23:42.725 INFO:tasks.cephadm:Deploying osd.7 on smithi181 with /dev/vg_nvme/lv_1... 2024-05-03T10:23:42.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-03T10:23:42.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:23:42 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:23:42] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:23:42.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: Adjusting osd_memory_target on smithi181 to 5354M 2024-05-03T10:23:42.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-03T10:23:42.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: osdmap e44: 7 total, 6 up, 7 in 2024-05-03T10:23:42.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='osd.6 [v2:172.21.15.181:6816/4115880036,v1:172.21.15.181:6817/4115880036]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: Adjusting osd_memory_target on smithi181 to 5354M 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-03T10:23:42.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: osdmap e44: 7 total, 6 up, 7 in 2024-05-03T10:23:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='osd.6 [v2:172.21.15.181:6816/4115880036,v1:172.21.15.181:6817/4115880036]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:42.794 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:42 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: Adjusting osd_memory_target on smithi181 to 5354M 2024-05-03T10:23:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-03T10:23:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: osdmap e44: 7 total, 6 up, 7 in 2024-05-03T10:23:42.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:42.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='osd.6 [v2:172.21.15.181:6816/4115880036,v1:172.21.15.181:6817/4115880036]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:42.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:23:42.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:42.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:42 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:42.872 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:23:42 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:23:42.807+0000 7f47eb683700 -1 osd.6 0 waiting for initial osdmap 2024-05-03T10:23:42.872 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:23:42 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:23:42.813+0000 7f47e4817700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T10:23:43.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:43 smithi181 ceph-mon[23393]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:43.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:43 smithi181 ceph-mon[23393]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:23:43.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:43 smithi181 ceph-mon[23393]: osdmap e45: 7 total, 6 up, 7 in 2024-05-03T10:23:43.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:43 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:43.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:43 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:44.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[22404]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:44.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[22404]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:23:44.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[22404]: osdmap e45: 7 total, 6 up, 7 in 2024-05-03T10:23:44.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:44.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:44.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[27243]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-03T10:23:44.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[27243]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:23:44.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[27243]: osdmap e45: 7 total, 6 up, 7 in 2024-05-03T10:23:44.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:44.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:43 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:45.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:44 smithi181 ceph-mon[23393]: purged_snaps scrub starts 2024-05-03T10:23:45.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:44 smithi181 ceph-mon[23393]: purged_snaps scrub ok 2024-05-03T10:23:45.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:44 smithi181 ceph-mon[23393]: osd.6 [v2:172.21.15.181:6816/4115880036,v1:172.21.15.181:6817/4115880036] boot 2024-05-03T10:23:45.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:44 smithi181 ceph-mon[23393]: osdmap e46: 7 total, 7 up, 7 in 2024-05-03T10:23:45.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:44 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:45.053 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:44 smithi181 ceph-mon[23393]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:23:45.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[22404]: purged_snaps scrub starts 2024-05-03T10:23:45.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[22404]: purged_snaps scrub ok 2024-05-03T10:23:45.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[22404]: osd.6 [v2:172.21.15.181:6816/4115880036,v1:172.21.15.181:6817/4115880036] boot 2024-05-03T10:23:45.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[22404]: osdmap e46: 7 total, 7 up, 7 in 2024-05-03T10:23:45.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:45.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[22404]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:23:45.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[27243]: purged_snaps scrub starts 2024-05-03T10:23:45.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[27243]: purged_snaps scrub ok 2024-05-03T10:23:45.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[27243]: osd.6 [v2:172.21.15.181:6816/4115880036,v1:172.21.15.181:6817/4115880036] boot 2024-05-03T10:23:45.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[27243]: osdmap e46: 7 total, 7 up, 7 in 2024-05-03T10:23:45.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:23:45.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:44 smithi043 ceph-mon[27243]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:23:45.930 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-03T10:23:45.930 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-03T10:23:45.930 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-05-03T10:23:45.930 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-05-03T10:23:45.930 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123688 s, 848 MB/s 2024-05-03T10:23:45.931 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-05-03T10:23:46.021 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:45 smithi043 ceph-mon[27243]: osdmap e47: 7 total, 7 up, 7 in 2024-05-03T10:23:46.022 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:45 smithi043 ceph-mon[22404]: osdmap e47: 7 total, 7 up, 7 in 2024-05-03T10:23:46.022 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:23:45 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:23:45] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:23:46.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:45 smithi181 ceph-mon[23393]: osdmap e47: 7 total, 7 up, 7 in 2024-05-03T10:23:46.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_1 2024-05-03T10:23:46.949 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: osdmap e48: 7 total, 7 up, 7 in 2024-05-03T10:23:46.949 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:46.949 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: Detected new or changed devices on smithi181 2024-05-03T10:23:46.949 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:46.949 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:46.950 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:46.950 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:46.950 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:46.950 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:23:46.950 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:46 smithi181 ceph-mon[23393]: osdmap e49: 7 total, 7 up, 7 in 2024-05-03T10:23:47.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: osdmap e48: 7 total, 7 up, 7 in 2024-05-03T10:23:47.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:47.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: Detected new or changed devices on smithi181 2024-05-03T10:23:47.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:47.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[22404]: osdmap e49: 7 total, 7 up, 7 in 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: osdmap e48: 7 total, 7 up, 7 in 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: Detected new or changed devices on smithi181 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:23:47.044 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:46 smithi043 ceph-mon[27243]: osdmap e49: 7 total, 7 up, 7 in 2024-05-03T10:23:48.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:23:47 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:23:47] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:23:48.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:48 smithi181 ceph-mon[23393]: osdmap e50: 7 total, 7 up, 7 in 2024-05-03T10:23:48.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:48.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:48.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:23:48.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:23:48.844 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:48 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:49.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[22404]: osdmap e50: 7 total, 7 up, 7 in 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[27243]: osdmap e50: 7 total, 7 up, 7 in 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T10:23:49.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:48 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:49.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:49 smithi181 ceph-mon[23393]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:23:49.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:49 smithi181 ceph-mon[23393]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:23:50.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:49 smithi043 ceph-mon[22404]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:23:50.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:49 smithi043 ceph-mon[22404]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:23:50.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:49 smithi043 ceph-mon[27243]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:23:50.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:49 smithi043 ceph-mon[27243]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:23:51.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:51 smithi181 ceph-mon[23393]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-05-03T10:23:51.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:51 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/519849853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8"}]: dispatch 2024-05-03T10:23:51.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:51 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8"}]: dispatch 2024-05-03T10:23:51.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:51 smithi181 ceph-mon[23393]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8"}]': finished 2024-05-03T10:23:51.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:51 smithi181 ceph-mon[23393]: osdmap e51: 8 total, 7 up, 8 in 2024-05-03T10:23:51.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:51 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:23:51.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:51 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/3681017780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:23:52.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[22404]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-05-03T10:23:52.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/519849853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8"}]: dispatch 2024-05-03T10:23:52.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8"}]: dispatch 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[22404]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8"}]': finished 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[22404]: osdmap e51: 8 total, 7 up, 8 in 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/3681017780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[27243]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/519849853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8"}]: dispatch 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8"}]: dispatch 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[27243]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8"}]': finished 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[27243]: osdmap e51: 8 total, 7 up, 8 in 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:23:52.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:51 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/3681017780' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T10:23:53.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:53 smithi181 ceph-mon[23393]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T10:23:54.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:53 smithi043 ceph-mon[22404]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T10:23:54.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:53 smithi043 ceph-mon[27243]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T10:23:55.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:55 smithi181 ceph-mon[23393]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-03T10:23:55.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:23:55.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:55 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:23:56.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:55 smithi043 ceph-mon[22404]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-03T10:23:56.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:23:56.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:55 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:23:56.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:55 smithi043 ceph-mon[27243]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-03T10:23:56.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:23:56.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:55 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:23:57.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:57 smithi181 ceph-mon[23393]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-03T10:23:57.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-03T10:23:57.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:57 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:58.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:57 smithi043 ceph-mon[27243]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-03T10:23:58.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-03T10:23:58.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:57 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:58.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:57 smithi043 ceph-mon[22404]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-03T10:23:58.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-03T10:23:58.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:57 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:23:58.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:58 smithi181 ceph-mon[23393]: Deploying daemon osd.7 on smithi181 2024-05-03T10:23:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:58 smithi043 ceph-mon[27243]: Deploying daemon osd.7 on smithi181 2024-05-03T10:23:59.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:58 smithi043 ceph-mon[22404]: Deploying daemon osd.7 on smithi181 2024-05-03T10:23:59.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:23:59 smithi181 ceph-mon[23393]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-03T10:24:00.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:23:59 smithi043 ceph-mon[27243]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-03T10:24:00.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:23:59 smithi043 ceph-mon[22404]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-03T10:24:01.852 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:01 smithi181 ceph-mon[23393]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:02.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:01 smithi043 ceph-mon[27243]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:02.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:01 smithi043 ceph-mon[22404]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:03.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:03 smithi181 ceph-mon[23393]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:03.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:03 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:03.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:03 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:03.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:03 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:03.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:03 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:03.799 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:03 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:04.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[22404]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:04.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:04.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:04.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:04.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:04.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:04.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[27243]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:04.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:04.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:04.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:04.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:04.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:03 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:05.834 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:05 smithi181 ceph-mon[23393]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:05.943 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:05 smithi043 ceph-mon[22404]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:05.943 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:05 smithi043 ceph-mon[27243]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:06.260 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:05 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:24:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: Detected new or changed devices on smithi181 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: from='osd.7 [v2:172.21.15.181:6824/3974891654,v1:172.21.15.181:6825/3974891654]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T10:24:07.242 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:06 smithi181 ceph-mon[23393]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:07.275 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 7 on host 'smithi181' 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: Detected new or changed devices on smithi181 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: from='osd.7 [v2:172.21.15.181:6824/3974891654,v1:172.21.15.181:6825/3974891654]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T10:24:07.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[22404]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: Detected new or changed devices on smithi181 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: from='osd.7 [v2:172.21.15.181:6824/3974891654,v1:172.21.15.181:6825/3974891654]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T10:24:07.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:06 smithi043 ceph-mon[27243]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:07.896 DEBUG:teuthology.orchestra.run.smithi181:osd.7> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.7.service 2024-05-03T10:24:07.899 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-05-03T10:24:07.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-03T10:24:08.163 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-03T10:24:08.163 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: osdmap e52: 8 total, 7 up, 8 in 2024-05-03T10:24:08.163 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:08.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: from='osd.7 [v2:172.21.15.181:6824/3974891654,v1:172.21.15.181:6825/3974891654]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:24:08.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:24:08.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:08.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:08.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:08.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:08.164 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:07 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:08.164 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:24:07 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:24:07.964+0000 7f85bed84700 -1 osd.7 0 waiting for initial osdmap 2024-05-03T10:24:08.164 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:24:07 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:24:07.972+0000 7f85b971b700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T10:24:08.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-03T10:24:08.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: osdmap e52: 8 total, 7 up, 8 in 2024-05-03T10:24:08.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:08.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: from='osd.7 [v2:172.21.15.181:6824/3974891654,v1:172.21.15.181:6825/3974891654]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:24:08.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:24:08.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:08.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:08.230 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:08.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:08.231 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:08.231 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:07 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:24:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:24:08.231 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:08 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:24:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:24:08.231 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-03T10:24:08.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: osdmap e52: 8 total, 7 up, 8 in 2024-05-03T10:24:08.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:08.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: from='osd.7 [v2:172.21.15.181:6824/3974891654,v1:172.21.15.181:6825/3974891654]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:24:08.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-05-03T10:24:08.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:08.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:08.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:08.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:08.232 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:07 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:24:09.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: osdmap e53: 8 total, 7 up, 8 in 2024-05-03T10:24:09.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:09.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:09.273 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: osdmap e53: 8 total, 7 up, 8 in 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:09.274 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:09.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:09.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:09.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:09.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:09.275 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:08 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-05-03T10:24:09.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: osdmap e53: 8 total, 7 up, 8 in 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:09.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:08 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:10.216 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[27243]: purged_snaps scrub starts 2024-05-03T10:24:10.216 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[27243]: purged_snaps scrub ok 2024-05-03T10:24:10.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[27243]: osd.7 [v2:172.21.15.181:6824/3974891654,v1:172.21.15.181:6825/3974891654] boot 2024-05-03T10:24:10.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[27243]: osdmap e54: 8 total, 8 up, 8 in 2024-05-03T10:24:10.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[22404]: purged_snaps scrub starts 2024-05-03T10:24:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[22404]: purged_snaps scrub ok 2024-05-03T10:24:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[22404]: osd.7 [v2:172.21.15.181:6824/3974891654,v1:172.21.15.181:6825/3974891654] boot 2024-05-03T10:24:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[22404]: osdmap e54: 8 total, 8 up, 8 in 2024-05-03T10:24:10.217 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:09 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:10.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:09 smithi181 ceph-mon[23393]: purged_snaps scrub starts 2024-05-03T10:24:10.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:09 smithi181 ceph-mon[23393]: purged_snaps scrub ok 2024-05-03T10:24:10.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:09 smithi181 ceph-mon[23393]: osd.7 [v2:172.21.15.181:6824/3974891654,v1:172.21.15.181:6825/3974891654] boot 2024-05-03T10:24:10.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:09 smithi181 ceph-mon[23393]: osdmap e54: 8 total, 8 up, 8 in 2024-05-03T10:24:10.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:09 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:10.668 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:24:11.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:10 smithi043 ceph-mon[22404]: osdmap e55: 8 total, 8 up, 8 in 2024-05-03T10:24:11.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:10 smithi043 ceph-mon[22404]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:11.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:10 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1486234802' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T10:24:11.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:10 smithi043 ceph-mon[27243]: osdmap e55: 8 total, 8 up, 8 in 2024-05-03T10:24:11.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:10 smithi043 ceph-mon[27243]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:11.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:10 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1486234802' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T10:24:11.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:10 smithi181 ceph-mon[23393]: osdmap e55: 8 total, 8 up, 8 in 2024-05-03T10:24:11.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:10 smithi181 ceph-mon[23393]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:11.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:10 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1486234802' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T10:24:11.408 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1714731848,"num_in_osds":8,"osd_in_since":1714731830,"num_remapped_pgs":1} 2024-05-03T10:24:11.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T10:24:11.649 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:12.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:11 smithi043 ceph-mon[27243]: osdmap e56: 8 total, 8 up, 8 in 2024-05-03T10:24:12.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:11 smithi043 ceph-mon[27243]: osdmap e57: 8 total, 8 up, 8 in 2024-05-03T10:24:12.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:11 smithi043 ceph-mon[22404]: osdmap e56: 8 total, 8 up, 8 in 2024-05-03T10:24:12.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:11 smithi043 ceph-mon[22404]: osdmap e57: 8 total, 8 up, 8 in 2024-05-03T10:24:12.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:11 smithi181 ceph-mon[23393]: osdmap e56: 8 total, 8 up, 8 in 2024-05-03T10:24:12.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:11 smithi181 ceph-mon[23393]: osdmap e57: 8 total, 8 up, 8 in 2024-05-03T10:24:13.478 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:24:13.478 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":58,"fsid":"79bfd44e-0936-11ef-bc97-c7b262605968","created":"2024-05-03T10:19:06.824252+0000","modified":"2024-05-03T10:24:12.607143+0000","last_up_change":"2024-05-03T10:24:08.961673+0000","last_in_change":"2024-05-03T10:23:50.804062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T10:22:07.541250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6db4ef97-76b0-4df2-9b32-106b07e97bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6803","nonce":2876312201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6805","nonce":2876312201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6809","nonce":2876312201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6807","nonce":2876312201}]},"public_addr":"172.21.15.43:6803/2876312201","cluster_addr":"172.21.15.43:6805/2876312201","heartbeat_back_addr":"172.21.15.43:6809/2876312201","heartbeat_front_addr":"172.21.15.43:6807/2876312201","state":["exists","up"]},{"osd":1,"uuid":"195e4d8e-8690-4122-845e-78c7ec670b64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6811","nonce":3187463390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6813","nonce":3187463390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6817","nonce":3187463390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6815","nonce":3187463390}]},"public_addr":"172.21.15.43:6811/3187463390","cluster_addr":"172.21.15.43:6813/3187463390","heartbeat_back_addr":"172.21.15.43:6817/3187463390","heartbeat_front_addr":"172.21.15.43:6815/3187463390","state":["exists","up"]},{"osd":2,"uuid":"05320fef-7e52-4c68-8484-317080e0c4bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6819","nonce":260846504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6821","nonce":260846504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6825","nonce":260846504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6823","nonce":260846504}]},"public_addr":"172.21.15.43:6819/260846504","cluster_addr":"172.21.15.43:6821/260846504","heartbeat_back_addr":"172.21.15.43:6825/260846504","heartbeat_front_addr":"172.21.15.43:6823/260846504","state":["exists","up"]},{"osd":3,"uuid":"013775c5-f256-4587-a80f-febbf5992d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6827","nonce":1619280720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6829","nonce":1619280720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6833","nonce":1619280720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6831","nonce":1619280720}]},"public_addr":"172.21.15.43:6827/1619280720","cluster_addr":"172.21.15.43:6829/1619280720","heartbeat_back_addr":"172.21.15.43:6833/1619280720","heartbeat_front_addr":"172.21.15.43:6831/1619280720","state":["exists","up"]},{"osd":4,"uuid":"60188084-c744-4a5a-8593-2fe14c1a3349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6801","nonce":3176358771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6803","nonce":3176358771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6807","nonce":3176358771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6805","nonce":3176358771}]},"public_addr":"172.21.15.181:6801/3176358771","cluster_addr":"172.21.15.181:6803/3176358771","heartbeat_back_addr":"172.21.15.181:6807/3176358771","heartbeat_front_addr":"172.21.15.181:6805/3176358771","state":["exists","up"]},{"osd":5,"uuid":"b6e7152a-0120-43d4-b099-38a03f9947b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6809","nonce":2900949392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6811","nonce":2900949392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6815","nonce":2900949392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6813","nonce":2900949392}]},"public_addr":"172.21.15.181:6809/2900949392","cluster_addr":"172.21.15.181:6811/2900949392","heartbeat_back_addr":"172.21.15.181:6815/2900949392","heartbeat_front_addr":"172.21.15.181:6813/2900949392","state":["exists","up"]},{"osd":6,"uuid":"85eb0fcf-b686-4e6c-a116-9dc7b55379ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6817","nonce":4115880036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6819","nonce":4115880036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6823","nonce":4115880036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6821","nonce":4115880036}]},"public_addr":"172.21.15.181:6817/4115880036","cluster_addr":"172.21.15.181:6819/4115880036","heartbeat_back_addr":"172.21.15.181:6823/4115880036","heartbeat_front_addr":"172.21.15.181:6821/4115880036","state":["exists","up"]},{"osd":7,"uuid":"7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6825","nonce":3974891654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6827","nonce":3974891654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6831","nonce":3974891654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6829","nonce":3974891654}]},"public_addr":"172.21.15.181:6825/3974891654","cluster_addr":"172.21.15.181:6827/3974891654","heartbeat_back_addr":"172.21.15.181:6831/3974891654","heartbeat_front_addr":"172.21.15.181:6829/3974891654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:21:17.474097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:21:40.691353+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:22:05.291973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:22:31.118802+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:22:55.282759+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:23:18.312154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:23:42.095639+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:24:07.044206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/3757009401":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/2216262703":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/418629400":"2024-05-04T10:19:54.983634+0000","172.21.15.43:6801/2358376213":"2024-05-04T10:19:54.983634+0000","172.21.15.43:6800/2358376213":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/3170124346":"2024-05-04T10:19:32.819179+0000","172.21.15.43:0/1256461377":"2024-05-04T10:19:32.819179+0000","172.21.15.43:6801/992331217":"2024-05-04T10:19:32.819179+0000","172.21.15.43:6800/992331217":"2024-05-04T10:19:32.819179+0000","172.21.15.43:0/868367844":"2024-05-04T10:19:32.819179+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T10:24:13.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:13 smithi043 ceph-mon[22404]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:13.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:13 smithi043 ceph-mon[22404]: osdmap e58: 8 total, 8 up, 8 in 2024-05-03T10:24:13.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:13 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2234385658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T10:24:13.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:13 smithi043 ceph-mon[27243]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:13.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:13 smithi043 ceph-mon[27243]: osdmap e58: 8 total, 8 up, 8 in 2024-05-03T10:24:13.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:13 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2234385658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T10:24:13.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:13 smithi181 ceph-mon[23393]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:13.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:13 smithi181 ceph-mon[23393]: osdmap e58: 8 total, 8 up, 8 in 2024-05-03T10:24:13.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:13 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2234385658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T10:24:14.130 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-03T10:22:07.541250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-05-03T10:24:14.130 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-03T10:24:14.354 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:15.736 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:15 smithi043 ceph-mon[22404]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:15.736 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:15 smithi043 ceph-mon[27243]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:15.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:15 smithi181 ceph-mon[23393]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:16.129 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-05-03T10:24:16.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:16 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2190934439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-03T10:24:16.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:16 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2190934439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-03T10:24:16.788 INFO:tasks.cephadm:Adding prometheus.a on smithi181 2024-05-03T10:24:16.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch apply prometheus '1;smithi181=a' 2024-05-03T10:24:16.812 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:16 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2190934439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-03T10:24:17.683 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:17 smithi181 ceph-mon[23393]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T10:24:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:17 smithi043 ceph-mon[22404]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T10:24:18.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:17 smithi043 ceph-mon[27243]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-03T10:24:18.610 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled prometheus update... 2024-05-03T10:24:19.206 DEBUG:teuthology.orchestra.run.smithi181:prometheus.a> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@prometheus.a.service 2024-05-03T10:24:19.248 INFO:tasks.cephadm:Adding node-exporter.a on smithi043 2024-05-03T10:24:19.248 INFO:tasks.cephadm:Adding node-exporter.b on smithi181 2024-05-03T10:24:19.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch apply node-exporter '2;smithi043=a;smithi181=b' 2024-05-03T10:24:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mon[23393]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-05-03T10:24:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mon[23393]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mon[23393]: Saving service prometheus spec with placement smithi181=a;count:1 2024-05-03T10:24:19.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:19.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:19.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:19.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:19.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:19.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-03T10:24:19.870 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ignoring --setuser ceph since I am not root 2024-05-03T10:24:19.870 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ignoring --setgroup ceph since I am not root 2024-05-03T10:24:19.870 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:19 smithi181 ceph-mgr[24909]: -- 172.21.15.181:0/3981836264 <== mon.2 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557772bb2340 con 0x557772b8a800 2024-05-03T10:24:19.870 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:19.813+0000 7fb266c82000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T10:24:19.960 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[27243]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-05-03T10:24:19.960 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[27243]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:19.960 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[27243]: Saving service prometheus spec with placement smithi181=a;count:1 2024-05-03T10:24:19.960 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[22404]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[22404]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[22404]: Saving service prometheus spec with placement smithi181=a;count:1 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:19.961 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:19.962 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' 2024-05-03T10:24:19.962 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-03T10:24:19.962 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:19 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ignoring --setuser ceph since I am not root 2024-05-03T10:24:19.962 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:19 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ignoring --setgroup ceph since I am not root 2024-05-03T10:24:19.962 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:19 smithi043 ceph-mgr[22638]: -- 172.21.15.43:0/2469923746 <== mon.0 v2:172.21.15.43:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560cbd27e340 con 0x560cbd256800 2024-05-03T10:24:19.962 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:19 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:19.825+0000 7f4269739000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T10:24:19.962 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:19 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:19.959+0000 7f4269739000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:24:20.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:20 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:20.061+0000 7f4269739000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T10:24:20.349 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:19.944+0000 7fb266c82000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:24:20.349 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:20.042+0000 7fb266c82000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T10:24:20.350 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:20.348+0000 7fb266c82000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T10:24:20.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:20.431+0000 7fb266c82000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T10:24:20.655 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:20 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:20.380+0000 7f4269739000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T10:24:20.655 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:20 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:20.467+0000 7f4269739000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T10:24:20.959 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:20 smithi181 ceph-mon[23393]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-03T10:24:20.959 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:20 smithi181 ceph-mon[23393]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-03T10:24:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:20 smithi043 ceph-mon[22404]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-03T10:24:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:20 smithi043 ceph-mon[22404]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-03T10:24:21.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:20 smithi043 ceph-mon[27243]: from='mgr.14152 172.21.15.43:0/3471412846' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-03T10:24:21.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:20 smithi043 ceph-mon[27243]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-03T10:24:21.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:21.579+0000 7fb266c82000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T10:24:21.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:21.741+0000 7fb266c82000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T10:24:21.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:21.821+0000 7fb266c82000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T10:24:21.913 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:21 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:21.654+0000 7f4269739000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T10:24:21.913 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:21 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:21.826+0000 7f4269739000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T10:24:22.235 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:22 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:22.138+0000 7fb266c82000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T10:24:22.250 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:21 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:21.912+0000 7f4269739000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T10:24:22.528 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:22 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:22.234+0000 7fb266c82000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T10:24:22.529 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:22 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:22.337+0000 7fb266c82000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T10:24:22.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:22 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:22.250+0000 7f4269739000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T10:24:22.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:22 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:22.359+0000 7f4269739000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T10:24:22.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:22 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:22.466+0000 7f4269739000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T10:24:22.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:22 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:22.591+0000 7fb266c82000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:24:23.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:22 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:22.718+0000 7f4269739000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T10:24:23.274 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:23 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:23.048+0000 7fb266c82000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T10:24:23.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:23.183+0000 7f4269739000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T10:24:23.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:23 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:23.551+0000 7fb266c82000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T10:24:23.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:23 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:23.654+0000 7fb266c82000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T10:24:24.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:23.710+0000 7f4269739000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T10:24:24.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:23.810+0000 7f4269739000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T10:24:24.361 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:23 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:23.997+0000 7fb266c82000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T10:24:24.361 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:24.090+0000 7fb266c82000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T10:24:24.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:24 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:24.120+0000 7f4269739000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T10:24:24.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:24 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:24.214+0000 7f4269739000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T10:24:24.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:24.514+0000 7fb266c82000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T10:24:24.941 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:24.646+0000 7fb266c82000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T10:24:24.941 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:24.739+0000 7fb266c82000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T10:24:24.941 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:24.849+0000 7fb266c82000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T10:24:24.941 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:24.939+0000 7fb266c82000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T10:24:24.996 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:24 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:24.637+0000 7f4269739000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T10:24:24.996 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:24 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:24.776+0000 7f4269739000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T10:24:24.996 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:24 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:24.875+0000 7f4269739000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T10:24:24.996 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:24 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:24.995+0000 7f4269739000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T10:24:25.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:25 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:25.089+0000 7f4269739000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T10:24:25.837 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:25.477+0000 7fb266c82000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T10:24:26.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:25 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:25.668+0000 7f4269739000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T10:24:26.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[27243]: Standby manager daemon x restarted 2024-05-03T10:24:26.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[27243]: Standby manager daemon x started 2024-05-03T10:24:26.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[27243]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-03T10:24:26.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[27243]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T10:24:26.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[27243]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-03T10:24:26.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[27243]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T10:24:26.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[22404]: Standby manager daemon x restarted 2024-05-03T10:24:26.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[22404]: Standby manager daemon x started 2024-05-03T10:24:26.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[22404]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-03T10:24:26.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[22404]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T10:24:26.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[22404]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-03T10:24:26.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:25 smithi043 ceph-mon[22404]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T10:24:26.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:25 smithi181 ceph-mon[23393]: Standby manager daemon x restarted 2024-05-03T10:24:26.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:25 smithi181 ceph-mon[23393]: Standby manager daemon x started 2024-05-03T10:24:26.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:25 smithi181 ceph-mon[23393]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-03T10:24:26.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:25 smithi181 ceph-mon[23393]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T10:24:26.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:25 smithi181 ceph-mon[23393]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-03T10:24:26.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:25 smithi181 ceph-mon[23393]: from='mgr.? 172.21.15.181:0/2454025788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T10:24:26.120 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:25.834+0000 7fb266c82000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T10:24:26.120 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: [03/May/2024:10:24:25] ENGINE Bus STARTING 2024-05-03T10:24:26.120 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: CherryPy Checker: 2024-05-03T10:24:26.120 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: The Application mounted at '' has an empty config. 2024-05-03T10:24:26.120 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: 2024-05-03T10:24:26.120 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: [03/May/2024:10:24:25] ENGINE Serving on http://:::9283 2024-05-03T10:24:26.120 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:24:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: [03/May/2024:10:24:25] ENGINE Bus STARTED 2024-05-03T10:24:26.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:26 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:26.043+0000 7f4269739000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T10:24:27.086 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:26 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:24:26] ENGINE Bus STARTING 2024-05-03T10:24:27.086 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:26 smithi043 ceph-mon[22404]: mgrmap e17: y(active, since 4m), standbys: x 2024-05-03T10:24:27.086 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:26 smithi043 ceph-mon[22404]: Active manager daemon y restarted 2024-05-03T10:24:27.086 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:26 smithi043 ceph-mon[22404]: Activating manager daemon y 2024-05-03T10:24:27.086 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:26 smithi043 ceph-mon[22404]: osdmap e59: 8 total, 8 up, 8 in 2024-05-03T10:24:27.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:26 smithi043 ceph-mon[27243]: mgrmap e17: y(active, since 4m), standbys: x 2024-05-03T10:24:27.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:26 smithi043 ceph-mon[27243]: Active manager daemon y restarted 2024-05-03T10:24:27.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:26 smithi043 ceph-mon[27243]: Activating manager daemon y 2024-05-03T10:24:27.087 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:26 smithi043 ceph-mon[27243]: osdmap e59: 8 total, 8 up, 8 in 2024-05-03T10:24:27.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:26 smithi181 ceph-mon[23393]: mgrmap e17: y(active, since 4m), standbys: x 2024-05-03T10:24:27.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:26 smithi181 ceph-mon[23393]: Active manager daemon y restarted 2024-05-03T10:24:27.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:26 smithi181 ceph-mon[23393]: Activating manager daemon y 2024-05-03T10:24:27.220 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:26 smithi181 ceph-mon[23393]: osdmap e59: 8 total, 8 up, 8 in 2024-05-03T10:24:27.336 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:27 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: CherryPy Checker: 2024-05-03T10:24:27.336 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:27 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: The Application mounted at '' has an empty config. 2024-05-03T10:24:27.336 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:27 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:24:27.336 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:27 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:24:27] ENGINE Serving on http://:::9283 2024-05-03T10:24:27.336 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:27 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:24:27] ENGINE Bus STARTED 2024-05-03T10:24:27.933 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled node-exporter update... 2024-05-03T10:24:28.159 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: mgrmap e18: y(active, starting, since 0.85584s), standbys: x 2024-05-03T10:24:28.159 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T10:24:28.160 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: Manager daemon y is now available 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]': finished 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]': finished 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: Updating smithi181:/etc/ceph/ceph.conf 2024-05-03T10:24:28.161 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-03T10:24:28.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:24:28.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:24:28.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:24:28.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:24:28.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.162 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-03T10:24:28.162 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: mgrmap e18: y(active, starting, since 0.85584s), standbys: x 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T10:24:28.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: Manager daemon y is now available 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]': finished 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]': finished 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: Updating smithi181:/etc/ceph/ceph.conf 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-03T10:24:28.164 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:24:28.165 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:24:28.165 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:24:28.165 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:24:28.165 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.165 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.165 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.165 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-03T10:24:28.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: mgrmap e18: y(active, starting, since 0.85584s), standbys: x 2024-05-03T10:24:28.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T10:24:28.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: Manager daemon y is now available 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi043", "name": "osd_memory_target"}]': finished 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]': finished 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: Updating smithi181:/etc/ceph/ceph.conf 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: Updating smithi043:/etc/ceph/ceph.conf 2024-05-03T10:24:28.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:24:28.372 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:24:28.372 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:24:28.372 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: Updating smithi043:/etc/ceph/ceph.client.admin.keyring 2024-05-03T10:24:28.372 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.372 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.372 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:28.372 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-03T10:24:28.501 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:28 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:24:28] ENGINE Bus STARTING 2024-05-03T10:24:28.501 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:28 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:24:28] ENGINE Serving on https://172.21.15.43:7150 2024-05-03T10:24:28.501 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:28 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: [03/May/2024:10:24:28] ENGINE Bus STARTED 2024-05-03T10:24:28.818 DEBUG:teuthology.orchestra.run.smithi043:node-exporter.a> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@node-exporter.a.service 2024-05-03T10:24:28.821 DEBUG:teuthology.orchestra.run.smithi181:node-exporter.b> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@node-exporter.b.service 2024-05-03T10:24:28.823 INFO:tasks.cephadm:Adding alertmanager.a on smithi043 2024-05-03T10:24:28.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch apply alertmanager '1;smithi043=a' 2024-05-03T10:24:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-03T10:24:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: mgrmap e19: y(active, since 1.87139s), standbys: x 2024-05-03T10:24:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi043=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: Saving service node-exporter spec with placement smithi043=a;smithi181=b;count:2 2024-05-03T10:24:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:29.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: Deploying daemon prometheus.a on smithi181 2024-05-03T10:24:29.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: [03/May/2024:10:24:28] ENGINE Bus STARTING 2024-05-03T10:24:29.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: [03/May/2024:10:24:28] ENGINE Serving on https://172.21.15.43:7150 2024-05-03T10:24:29.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: [03/May/2024:10:24:28] ENGINE Bus STARTED 2024-05-03T10:24:29.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:29.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-03T10:24:29.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: mgrmap e19: y(active, since 1.87139s), standbys: x 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi043=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: Saving service node-exporter spec with placement smithi043=a;smithi181=b;count:2 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: Deploying daemon prometheus.a on smithi181 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: [03/May/2024:10:24:28] ENGINE Bus STARTING 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: [03/May/2024:10:24:28] ENGINE Serving on https://172.21.15.43:7150 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: [03/May/2024:10:24:28] ENGINE Bus STARTED 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: mgrmap e19: y(active, since 1.87139s), standbys: x 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi043=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:29.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: Saving service node-exporter spec with placement smithi043=a;smithi181=b;count:2 2024-05-03T10:24:29.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:29.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:29.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: Deploying daemon prometheus.a on smithi181 2024-05-03T10:24:29.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: [03/May/2024:10:24:28] ENGINE Bus STARTING 2024-05-03T10:24:29.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: [03/May/2024:10:24:28] ENGINE Serving on https://172.21.15.43:7150 2024-05-03T10:24:29.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: [03/May/2024:10:24:28] ENGINE Bus STARTED 2024-05-03T10:24:29.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:30.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:29 smithi043 ceph-mon[27243]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:30.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:29 smithi043 ceph-mon[27243]: mgrmap e20: y(active, since 2s), standbys: x 2024-05-03T10:24:30.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:29 smithi043 ceph-mon[22404]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:30.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:29 smithi043 ceph-mon[22404]: mgrmap e20: y(active, since 2s), standbys: x 2024-05-03T10:24:30.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:29 smithi181 ceph-mon[23393]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:30.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:29 smithi181 ceph-mon[23393]: mgrmap e20: y(active, since 2s), standbys: x 2024-05-03T10:24:31.307 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled alertmanager update... 2024-05-03T10:24:32.337 DEBUG:teuthology.orchestra.run.smithi043:alertmanager.a> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@alertmanager.a.service 2024-05-03T10:24:32.341 INFO:tasks.cephadm:Adding grafana.a on smithi181 2024-05-03T10:24:32.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph orch apply grafana '1;smithi181=a' 2024-05-03T10:24:32.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:32 smithi181 ceph-mon[23393]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:32.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:32 smithi181 ceph-mon[23393]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi043=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:32.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:32 smithi181 ceph-mon[23393]: Saving service alertmanager spec with placement smithi043=a;count:1 2024-05-03T10:24:32.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:32 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:32.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:32 smithi043 ceph-mon[27243]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:32.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:32 smithi043 ceph-mon[27243]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi043=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:32.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:32 smithi043 ceph-mon[27243]: Saving service alertmanager spec with placement smithi043=a;count:1 2024-05-03T10:24:32.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:32 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:32.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:32 smithi043 ceph-mon[22404]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:32.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:32 smithi043 ceph-mon[22404]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi043=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:32.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:32 smithi043 ceph-mon[22404]: Saving service alertmanager spec with placement smithi043=a;count:1 2024-05-03T10:24:32.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:32 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:34.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:34 smithi181 ceph-mon[23393]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:34.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:34 smithi043 ceph-mon[22404]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:34.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:34 smithi043 ceph-mon[27243]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:35.315 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled grafana update... 2024-05-03T10:24:36.580 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:36 smithi181 ceph-mon[23393]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:36.581 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:36 smithi181 ceph-mon[23393]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:36.581 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:36 smithi181 ceph-mon[23393]: Saving service grafana spec with placement smithi181=a;count:1 2024-05-03T10:24:36.581 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:36 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:36.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:36 smithi043 ceph-mon[22404]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:36.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:36 smithi043 ceph-mon[22404]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:36.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:36 smithi043 ceph-mon[22404]: Saving service grafana spec with placement smithi181=a;count:1 2024-05-03T10:24:36.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:36 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:36.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:36 smithi043 ceph-mon[27243]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:36.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:36 smithi043 ceph-mon[27243]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:24:36.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:36 smithi043 ceph-mon[27243]: Saving service grafana spec with placement smithi181=a;count:1 2024-05-03T10:24:36.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:36 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:37.455 DEBUG:teuthology.orchestra.run.smithi181:grafana.a> sudo journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@grafana.a.service 2024-05-03T10:24:37.458 INFO:tasks.cephadm:Setting up client nodes... 2024-05-03T10:24:37.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-03T10:24:38.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:38 smithi181 ceph-mon[23393]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:38.896 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:38 smithi043 ceph-mon[27243]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:38.896 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:38 smithi043 ceph-mon[22404]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:39.670 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-05-03T10:24:39.670 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBnuzRmAGyrJxAAPjNb4qSvGxf7txjheRY9tQ== 2024-05-03T10:24:40.448 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-05-03T10:24:40.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-03T10:24:40.448 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-03T10:24:40.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-03T10:24:40.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:40 smithi181 ceph-mon[23393]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:40.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:40 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/4148363571' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T10:24:40.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:40 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/4148363571' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T10:24:40.926 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:40 smithi043 ceph-mon[27243]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:40.926 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:40 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/4148363571' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T10:24:40.926 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:40 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/4148363571' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T10:24:40.926 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:40 smithi043 ceph-mon[22404]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:40.927 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:40 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/4148363571' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T10:24:40.927 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:40 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/4148363571' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T10:24:42.007 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:41 smithi181 systemd[1]: Starting Ceph prometheus.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:24:42.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:42 smithi043 ceph-mon[22404]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:42.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:42 smithi043 ceph-mon[27243]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:42 smithi181 ceph-mon[23393]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:44.279 INFO:teuthology.orchestra.run.smithi181.stdout:[client.1] 2024-05-03T10:24:44.280 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBsuzRmpFhREBAAbPvNa8G1c423LFi2ACiqxw== 2024-05-03T10:24:44.367 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:44 smithi181 podman[39712]: 2024-05-03 10:24:44.091102321 +0000 UTC m=+0.549918211 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-03T10:24:44.367 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:44 smithi181 podman[39712]: 2024-05-03 10:24:44.22517196 +0000 UTC m=+0.683987847 container create 306833a359290518403be9c77ba9e00a9ac1d489fa98564ed0ac80d4bb958aef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:24:44.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:44 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:24:44] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:24:44.947 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:44 smithi181 ceph-mon[23393]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:44.948 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:44 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:44.948 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:44 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:44.948 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:44 smithi181 ceph-mon[23393]: from='client.? 172.21.15.181:0/2381080816' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T10:24:44.948 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:44 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T10:24:44.948 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:44 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T10:24:44.973 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-05-03T10:24:44.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-03T10:24:44.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-03T10:24:45.013 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-03T10:24:45.014 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-03T10:24:45.014 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph mgr dump --format=json 2024-05-03T10:24:45.041 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[22404]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[22404]: from='client.? 172.21.15.181:0/2381080816' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T10:24:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T10:24:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T10:24:45.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[27243]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:45.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:45.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:45.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[27243]: from='client.? 172.21.15.181:0/2381080816' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T10:24:45.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T10:24:45.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:44 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 podman[39712]: 2024-05-03 10:24:45.106983381 +0000 UTC m=+1.565799268 container init 306833a359290518403be9c77ba9e00a9ac1d489fa98564ed0ac80d4bb958aef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 podman[39712]: 2024-05-03 10:24:45.109866753 +0000 UTC m=+1.568682638 container start 306833a359290518403be9c77ba9e00a9ac1d489fa98564ed0ac80d4bb958aef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.133Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.133Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.133Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.133Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi181 (none))" 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.133Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.133Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.134Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-03T10:24:45.223 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.135Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.135Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.137Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.137Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.557µs 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.137Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.137Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.137Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.686µs wal_replay_duration=113.236µs total_replay_duration=144.54µs 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.138Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.138Z caller=main.go:947 level=info msg="TSDB started" 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.138Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.153Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.393232ms db_storage=850ns remote_storage=1.008µs web_handler=229ns query_engine=423ns scrape=331.028µs scrape_sd=14.789µs notify=727ns notify_sd=1.658µs rules=14.88439ms 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:24:45.153Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 bash[39712]: 306833a359290518403be9c77ba9e00a9ac1d489fa98564ed0ac80d4bb958aef 2024-05-03T10:24:45.224 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:24:45 smithi181 systemd[1]: Started Ceph prometheus.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:24:45.256 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:46.498 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.181:9095"}]: dispatch 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.181:9095"}]: dispatch 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:46.499 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:24:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:46 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:24:46.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:46.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.181:9095"}]: dispatch 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.181:9095"}]: dispatch 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:46.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:46.544 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.181:9095"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.181:9095"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:46.545 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:46.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:46.546 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:46 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:24:47.246 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:24:47.543 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:47 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:24:47] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:24:47.913 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":20,"active_gid":24328,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":3017762733},{"type":"v1","addr":"172.21.15.43:6801","nonce":3017762733}]},"active_addr":"172.21.15.43:6801/3017762733","active_change":"2024-05-03T10:24:26.046377+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24347,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.43:8443/","prometheus":"http://172.21.15.43:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":4135803371}]},{"addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":2428457514}]},{"addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":352714903}]},{"addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":2364900707}]}]}} 2024-05-03T10:24:47.919 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-03T10:24:47.919 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-03T10:24:47.919 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T10:24:48.080 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:47 smithi181 ceph-mon[23393]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:48.080 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:48.080 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:47 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/184161080' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T10:24:48.080 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:48.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:47 smithi043 ceph-mon[27243]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:48.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:48.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:47 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/184161080' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T10:24:48.163 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:48.164 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:47 smithi043 ceph-mon[22404]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:48.164 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:48.164 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:47 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/184161080' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T10:24:48.164 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:48.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:48 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:24:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:24:48.632 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:49.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:48 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:24:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:24:49.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:49 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:49 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:49 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.200 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:49 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.201 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:49 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:49.201 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:49 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:49.201 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:49 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:49.201 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:49 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.201 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:49 smithi181 ceph-mon[23393]: Deploying daemon node-exporter.a on smithi043 2024-05-03T10:24:49.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.489 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[22404]: Deploying daemon node-exporter.a on smithi043 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.490 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:24:49.491 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:24:49.491 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:24:49.491 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:49.491 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:49 smithi043 ceph-mon[27243]: Deploying daemon node-exporter.a on smithi043 2024-05-03T10:24:50.705 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:24:50 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:24:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:24:50.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:50 smithi181 ceph-mon[23393]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:50.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:50.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:50 smithi043 ceph-mon[22404]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:51.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:50 smithi043 ceph-mon[27243]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:51.758 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:24:51.759 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":59,"fsid":"79bfd44e-0936-11ef-bc97-c7b262605968","created":"2024-05-03T10:19:06.824252+0000","modified":"2024-05-03T10:24:26.045953+0000","last_up_change":"2024-05-03T10:24:08.961673+0000","last_in_change":"2024-05-03T10:23:50.804062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T10:22:07.541250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6db4ef97-76b0-4df2-9b32-106b07e97bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6803","nonce":2876312201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6805","nonce":2876312201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6809","nonce":2876312201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6807","nonce":2876312201}]},"public_addr":"172.21.15.43:6803/2876312201","cluster_addr":"172.21.15.43:6805/2876312201","heartbeat_back_addr":"172.21.15.43:6809/2876312201","heartbeat_front_addr":"172.21.15.43:6807/2876312201","state":["exists","up"]},{"osd":1,"uuid":"195e4d8e-8690-4122-845e-78c7ec670b64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6811","nonce":3187463390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6813","nonce":3187463390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6817","nonce":3187463390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6815","nonce":3187463390}]},"public_addr":"172.21.15.43:6811/3187463390","cluster_addr":"172.21.15.43:6813/3187463390","heartbeat_back_addr":"172.21.15.43:6817/3187463390","heartbeat_front_addr":"172.21.15.43:6815/3187463390","state":["exists","up"]},{"osd":2,"uuid":"05320fef-7e52-4c68-8484-317080e0c4bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6819","nonce":260846504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6821","nonce":260846504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6825","nonce":260846504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6823","nonce":260846504}]},"public_addr":"172.21.15.43:6819/260846504","cluster_addr":"172.21.15.43:6821/260846504","heartbeat_back_addr":"172.21.15.43:6825/260846504","heartbeat_front_addr":"172.21.15.43:6823/260846504","state":["exists","up"]},{"osd":3,"uuid":"013775c5-f256-4587-a80f-febbf5992d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6827","nonce":1619280720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6829","nonce":1619280720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6833","nonce":1619280720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6831","nonce":1619280720}]},"public_addr":"172.21.15.43:6827/1619280720","cluster_addr":"172.21.15.43:6829/1619280720","heartbeat_back_addr":"172.21.15.43:6833/1619280720","heartbeat_front_addr":"172.21.15.43:6831/1619280720","state":["exists","up"]},{"osd":4,"uuid":"60188084-c744-4a5a-8593-2fe14c1a3349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6801","nonce":3176358771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6803","nonce":3176358771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6807","nonce":3176358771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6805","nonce":3176358771}]},"public_addr":"172.21.15.181:6801/3176358771","cluster_addr":"172.21.15.181:6803/3176358771","heartbeat_back_addr":"172.21.15.181:6807/3176358771","heartbeat_front_addr":"172.21.15.181:6805/3176358771","state":["exists","up"]},{"osd":5,"uuid":"b6e7152a-0120-43d4-b099-38a03f9947b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6809","nonce":2900949392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6811","nonce":2900949392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6815","nonce":2900949392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6813","nonce":2900949392}]},"public_addr":"172.21.15.181:6809/2900949392","cluster_addr":"172.21.15.181:6811/2900949392","heartbeat_back_addr":"172.21.15.181:6815/2900949392","heartbeat_front_addr":"172.21.15.181:6813/2900949392","state":["exists","up"]},{"osd":6,"uuid":"85eb0fcf-b686-4e6c-a116-9dc7b55379ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6817","nonce":4115880036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6819","nonce":4115880036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6823","nonce":4115880036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6821","nonce":4115880036}]},"public_addr":"172.21.15.181:6817/4115880036","cluster_addr":"172.21.15.181:6819/4115880036","heartbeat_back_addr":"172.21.15.181:6823/4115880036","heartbeat_front_addr":"172.21.15.181:6821/4115880036","state":["exists","up"]},{"osd":7,"uuid":"7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6825","nonce":3974891654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6827","nonce":3974891654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6831","nonce":3974891654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6829","nonce":3974891654}]},"public_addr":"172.21.15.181:6825/3974891654","cluster_addr":"172.21.15.181:6827/3974891654","heartbeat_back_addr":"172.21.15.181:6831/3974891654","heartbeat_front_addr":"172.21.15.181:6829/3974891654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:21:17.474097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:21:40.691353+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:22:05.291973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:22:31.118802+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:22:55.282759+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:23:18.312154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:23:42.095639+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:24:07.044206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/2867505315":"2024-05-04T10:24:26.045904+0000","172.21.15.43:6801/2999399722":"2024-05-04T10:24:26.045904+0000","172.21.15.43:0/3757009401":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/3299376245":"2024-05-04T10:24:26.045904+0000","172.21.15.43:6800/2999399722":"2024-05-04T10:24:26.045904+0000","172.21.15.43:0/2216262703":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/418629400":"2024-05-04T10:19:54.983634+0000","172.21.15.43:6801/2358376213":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/1017574898":"2024-05-04T10:24:26.045904+0000","172.21.15.43:6800/2358376213":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/3170124346":"2024-05-04T10:19:32.819179+0000","172.21.15.43:0/1256461377":"2024-05-04T10:19:32.819179+0000","172.21.15.43:6801/992331217":"2024-05-04T10:19:32.819179+0000","172.21.15.43:6800/992331217":"2024-05-04T10:19:32.819179+0000","172.21.15.43:0/1526951672":"2024-05-04T10:24:26.045904+0000","172.21.15.43:0/868367844":"2024-05-04T10:19:32.819179+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T10:24:52.514 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-03T10:24:52.514 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T10:24:52.747 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:52.780 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:52 smithi043 ceph-mon[22404]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:52.781 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:52 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1689047328' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T10:24:52.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:52 smithi043 ceph-mon[27243]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:52.781 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:52 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1689047328' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T10:24:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:52 smithi181 ceph-mon[23393]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:52 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1689047328' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T10:24:54.549 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:24:54.549 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":59,"fsid":"79bfd44e-0936-11ef-bc97-c7b262605968","created":"2024-05-03T10:19:06.824252+0000","modified":"2024-05-03T10:24:26.045953+0000","last_up_change":"2024-05-03T10:24:08.961673+0000","last_in_change":"2024-05-03T10:23:50.804062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T10:22:07.541250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6db4ef97-76b0-4df2-9b32-106b07e97bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6803","nonce":2876312201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6805","nonce":2876312201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6809","nonce":2876312201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":2876312201},{"type":"v1","addr":"172.21.15.43:6807","nonce":2876312201}]},"public_addr":"172.21.15.43:6803/2876312201","cluster_addr":"172.21.15.43:6805/2876312201","heartbeat_back_addr":"172.21.15.43:6809/2876312201","heartbeat_front_addr":"172.21.15.43:6807/2876312201","state":["exists","up"]},{"osd":1,"uuid":"195e4d8e-8690-4122-845e-78c7ec670b64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6811","nonce":3187463390}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6813","nonce":3187463390}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6817","nonce":3187463390}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":3187463390},{"type":"v1","addr":"172.21.15.43:6815","nonce":3187463390}]},"public_addr":"172.21.15.43:6811/3187463390","cluster_addr":"172.21.15.43:6813/3187463390","heartbeat_back_addr":"172.21.15.43:6817/3187463390","heartbeat_front_addr":"172.21.15.43:6815/3187463390","state":["exists","up"]},{"osd":2,"uuid":"05320fef-7e52-4c68-8484-317080e0c4bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6819","nonce":260846504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6821","nonce":260846504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6825","nonce":260846504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":260846504},{"type":"v1","addr":"172.21.15.43:6823","nonce":260846504}]},"public_addr":"172.21.15.43:6819/260846504","cluster_addr":"172.21.15.43:6821/260846504","heartbeat_back_addr":"172.21.15.43:6825/260846504","heartbeat_front_addr":"172.21.15.43:6823/260846504","state":["exists","up"]},{"osd":3,"uuid":"013775c5-f256-4587-a80f-febbf5992d1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6826","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6827","nonce":1619280720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6828","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6829","nonce":1619280720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6832","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6833","nonce":1619280720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6830","nonce":1619280720},{"type":"v1","addr":"172.21.15.43:6831","nonce":1619280720}]},"public_addr":"172.21.15.43:6827/1619280720","cluster_addr":"172.21.15.43:6829/1619280720","heartbeat_back_addr":"172.21.15.43:6833/1619280720","heartbeat_front_addr":"172.21.15.43:6831/1619280720","state":["exists","up"]},{"osd":4,"uuid":"60188084-c744-4a5a-8593-2fe14c1a3349","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6801","nonce":3176358771}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6803","nonce":3176358771}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6807","nonce":3176358771}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":3176358771},{"type":"v1","addr":"172.21.15.181:6805","nonce":3176358771}]},"public_addr":"172.21.15.181:6801/3176358771","cluster_addr":"172.21.15.181:6803/3176358771","heartbeat_back_addr":"172.21.15.181:6807/3176358771","heartbeat_front_addr":"172.21.15.181:6805/3176358771","state":["exists","up"]},{"osd":5,"uuid":"b6e7152a-0120-43d4-b099-38a03f9947b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6809","nonce":2900949392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6811","nonce":2900949392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6815","nonce":2900949392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2900949392},{"type":"v1","addr":"172.21.15.181:6813","nonce":2900949392}]},"public_addr":"172.21.15.181:6809/2900949392","cluster_addr":"172.21.15.181:6811/2900949392","heartbeat_back_addr":"172.21.15.181:6815/2900949392","heartbeat_front_addr":"172.21.15.181:6813/2900949392","state":["exists","up"]},{"osd":6,"uuid":"85eb0fcf-b686-4e6c-a116-9dc7b55379ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6817","nonce":4115880036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6819","nonce":4115880036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6823","nonce":4115880036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":4115880036},{"type":"v1","addr":"172.21.15.181:6821","nonce":4115880036}]},"public_addr":"172.21.15.181:6817/4115880036","cluster_addr":"172.21.15.181:6819/4115880036","heartbeat_back_addr":"172.21.15.181:6823/4115880036","heartbeat_front_addr":"172.21.15.181:6821/4115880036","state":["exists","up"]},{"osd":7,"uuid":"7cd40dee-80f5-4dd9-8231-bb85cc8f1ac8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6825","nonce":3974891654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6827","nonce":3974891654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6831","nonce":3974891654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3974891654},{"type":"v1","addr":"172.21.15.181:6829","nonce":3974891654}]},"public_addr":"172.21.15.181:6825/3974891654","cluster_addr":"172.21.15.181:6827/3974891654","heartbeat_back_addr":"172.21.15.181:6831/3974891654","heartbeat_front_addr":"172.21.15.181:6829/3974891654","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:21:17.474097+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:21:40.691353+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:22:05.291973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:22:31.118802+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:22:55.282759+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:23:18.312154+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:23:42.095639+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T10:24:07.044206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.43:0/2867505315":"2024-05-04T10:24:26.045904+0000","172.21.15.43:6801/2999399722":"2024-05-04T10:24:26.045904+0000","172.21.15.43:0/3757009401":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/3299376245":"2024-05-04T10:24:26.045904+0000","172.21.15.43:6800/2999399722":"2024-05-04T10:24:26.045904+0000","172.21.15.43:0/2216262703":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/418629400":"2024-05-04T10:19:54.983634+0000","172.21.15.43:6801/2358376213":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/1017574898":"2024-05-04T10:24:26.045904+0000","172.21.15.43:6800/2358376213":"2024-05-04T10:19:54.983634+0000","172.21.15.43:0/3170124346":"2024-05-04T10:19:32.819179+0000","172.21.15.43:0/1256461377":"2024-05-04T10:19:32.819179+0000","172.21.15.43:6801/992331217":"2024-05-04T10:19:32.819179+0000","172.21.15.43:6800/992331217":"2024-05-04T10:19:32.819179+0000","172.21.15.43:0/1526951672":"2024-05-04T10:24:26.045904+0000","172.21.15.43:0/868367844":"2024-05-04T10:19:32.819179+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T10:24:54.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:54 smithi043 ceph-mon[22404]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:54.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:54 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3088853379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T10:24:54.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:54 smithi043 ceph-mon[27243]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:54.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:54 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3088853379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T10:24:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:54 smithi181 ceph-mon[23393]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:54 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3088853379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T10:24:55.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-03T10:24:55.371 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-03T10:24:55.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-03T10:24:55.372 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-03T10:24:55.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-03T10:24:55.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-03T10:24:55.373 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-03T10:24:55.374 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-03T10:24:56.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:55 smithi043 ceph-mon[22404]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:56.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:55 smithi043 ceph-mon[27243]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:56.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:55 smithi181 ceph-mon[23393]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:56.407 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:24:57.542 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 podman[43464]: 2024-05-03 10:24:57.446135641 +0000 UTC m=+3.082331872 container create 695e77613650d2a7823fc511f1a0516bedcf5dcd4320af6678a202be06b387a0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-03T10:24:58.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 podman[43464]: 2024-05-03 10:24:57.656682191 +0000 UTC m=+3.292878425 container init 695e77613650d2a7823fc511f1a0516bedcf5dcd4320af6678a202be06b387a0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-03T10:24:58.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 podman[43464]: 2024-05-03 10:24:57.66030833 +0000 UTC m=+3.296504562 container start 695e77613650d2a7823fc511f1a0516bedcf5dcd4320af6678a202be06b387a0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-03T10:24:58.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.665Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-03T10:24:58.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.665Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-03T10:24:58.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.665Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-03T10:24:58.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.665Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-03T10:24:58.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=arp 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=edac 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-03T10:24:58.044 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=os 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-03T10:24:58.045 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.666Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=stat 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=time 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=uname 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-03T10:24:58.046 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-03T10:24:58.047 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-03T10:24:58.047 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-03T10:24:58.047 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a[43969]: ts=2024-05-03T10:24:57.667Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-03T10:24:58.047 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 bash[43464]: 695e77613650d2a7823fc511f1a0516bedcf5dcd4320af6678a202be06b387a0 2024-05-03T10:24:58.047 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:24:57 smithi043 systemd[1]: Started Ceph node-exporter.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:24:58.738 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:58 smithi181 ceph-mon[23393]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:58.738 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:58 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:58.976 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:58 smithi043 ceph-mon[27243]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:58.976 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:58 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:58.976 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:58 smithi043 ceph-mon[22404]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:24:58.977 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:58 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:24:59.119 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:24:58 smithi181 bash[40603]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-05-03T10:24:59.619 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:24:59 smithi181 bash[40603]: Getting image source signatures 2024-05-03T10:24:59.619 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:24:59 smithi181 bash[40603]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-05-03T10:24:59.619 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:24:59 smithi181 bash[40603]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-05-03T10:24:59.619 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:24:59 smithi181 bash[40603]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-05-03T10:24:59.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:24:59 smithi043 ceph-mon[27243]: Deploying daemon node-exporter.b on smithi181 2024-05-03T10:24:59.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:24:59 smithi043 ceph-mon[22404]: Deploying daemon node-exporter.b on smithi181 2024-05-03T10:25:00.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:24:59 smithi181 ceph-mon[23393]: Deploying daemon node-exporter.b on smithi181 2024-05-03T10:25:00.892 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:00 smithi043 ceph-mon[27243]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:00.892 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:00 smithi043 ceph-mon[22404]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:00 smithi181 ceph-mon[23393]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:01.429 INFO:teuthology.orchestra.run.smithi043.stdout:55834574889 2024-05-03T10:25:01.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-03T10:25:01.619 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 bash[40603]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-05-03T10:25:01.619 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 bash[40603]: Writing manifest to image destination 2024-05-03T10:25:01.949 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 podman[40603]: 2024-05-03 10:25:01.587917701 +0000 UTC m=+2.947877686 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-03T10:25:01.949 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 podman[40603]: 2024-05-03 10:25:01.6952626 +0000 UTC m=+3.055222651 container create dbf7b3ccb3e232157aebda6dd64e8aa33c7b0c29a2a28101b4089726fbae9c6d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-03T10:25:01.949 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 podman[40603]: 2024-05-03 10:25:01.880742217 +0000 UTC m=+3.240702207 container init dbf7b3ccb3e232157aebda6dd64e8aa33c7b0c29a2a28101b4089726fbae9c6d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-03T10:25:01.949 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 podman[40603]: 2024-05-03 10:25:01.883282227 +0000 UTC m=+3.243242212 container start dbf7b3ccb3e232157aebda6dd64e8aa33c7b0c29a2a28101b4089726fbae9c6d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-03T10:25:01.949 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.887Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-03T10:25:01.949 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.887Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-03T10:25:01.949 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.887Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-03T10:25:01.950 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.887Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-03T10:25:01.950 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-03T10:25:01.950 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=arp 2024-05-03T10:25:01.950 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-03T10:25:01.950 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-03T10:25:01.950 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-03T10:25:01.950 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-03T10:25:01.950 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-03T10:25:01.950 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=edac 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-03T10:25:01.951 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=os 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-03T10:25:01.952 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=stat 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=time 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=uname 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-03T10:25:01.953 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-03T10:25:01.954 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b[40653]: ts=2024-05-03T10:25:01.888Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-03T10:25:01.954 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 bash[40603]: dbf7b3ccb3e232157aebda6dd64e8aa33c7b0c29a2a28101b4089726fbae9c6d 2024-05-03T10:25:02.369 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:25:01 smithi181 systemd[1]: Started Ceph node-exporter.b for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:25:02.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:02 smithi043 ceph-mon[22404]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:02.991 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:02.992 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:02 smithi043 ceph-mon[27243]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:02.992 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:02 smithi181 ceph-mon[23393]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:03.925 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:03 smithi043 ceph-mon[27243]: Deploying daemon alertmanager.a on smithi043 2024-05-03T10:25:03.925 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:03 smithi043 ceph-mon[22404]: Deploying daemon alertmanager.a on smithi043 2024-05-03T10:25:04.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:03 smithi181 ceph-mon[23393]: Deploying daemon alertmanager.a on smithi043 2024-05-03T10:25:04.739 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:04 smithi181 ceph-mon[23393]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:04.937 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:04 smithi043 ceph-mon[22404]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:04.938 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:04 smithi043 ceph-mon[27243]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:06.809 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:06 smithi181 ceph-mon[23393]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:07.017 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:06 smithi043 ceph-mon[27243]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:07.018 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:06 smithi043 ceph-mon[22404]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:07.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:07 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:25:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:08.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:07 smithi181 ceph-mon[23393]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:08.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:07 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:08.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:07 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:08.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:07 smithi043 ceph-mon[27243]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:08.217 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:07 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:08.218 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:07 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:08.218 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:07 smithi043 ceph-mon[22404]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:08.218 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:07 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:08.218 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:07 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:08.504 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:08 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:25:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:09.254 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:08 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:25:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:09.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:09 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:09.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:09 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:09.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:09 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:09.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:09 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:09.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:09 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:09.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:09 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:09.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:09 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:09.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:09 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:09.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:09 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:10.403 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:10 smithi043 ceph-mon[22404]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:10.404 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:10 smithi043 ceph-mon[27243]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:10.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:10 smithi181 ceph-mon[23393]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:11.124 INFO:teuthology.orchestra.run.smithi043.stdout:163208757271 2024-05-03T10:25:11.124 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-03T10:25:12.906 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:12 smithi043 ceph-mon[22404]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:12.906 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:12 smithi043 ceph-mon[27243]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:12 smithi181 ceph-mon[23393]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:14.023 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:14.182 INFO:teuthology.orchestra.run.smithi043.stdout:77309411366 2024-05-03T10:25:14.182 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-03T10:25:14.987 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:14 smithi043 ceph-mon[22404]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:14.987 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:14 smithi043 ceph-mon[27243]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:14 smithi181 ceph-mon[23393]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:15.850 INFO:teuthology.orchestra.run.smithi043.stdout:197568495634 2024-05-03T10:25:15.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-03T10:25:16.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:16 smithi043 ceph-mon[22404]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:16.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:16 smithi043 ceph-mon[27243]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:16.960 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:16 smithi181 ceph-mon[23393]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:18.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:18 smithi043 ceph-mon[22404]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:18.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:18 smithi043 ceph-mon[27243]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:18 smithi181 ceph-mon[23393]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:19.510 INFO:teuthology.orchestra.run.smithi043.stdout:34359738417 2024-05-03T10:25:19.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-03T10:25:20.156 INFO:teuthology.orchestra.run.smithi043.stdout:128849018909 2024-05-03T10:25:20.156 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-03T10:25:20.916 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:20 smithi043 ceph-mon[22404]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:20.917 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:20 smithi043 ceph-mon[27243]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:20 smithi181 ceph-mon[23393]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:21.560 INFO:teuthology.orchestra.run.smithi043.stdout:231928233999 2024-05-03T10:25:21.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-03T10:25:21.564 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:22.846 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:22 smithi043 ceph-mon[27243]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:22.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:22 smithi043 ceph-mon[22404]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:22 smithi181 ceph-mon[23393]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:23.148 INFO:teuthology.orchestra.run.smithi043.stdout:107374182436 2024-05-03T10:25:23.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-03T10:25:23.828 INFO:teuthology.orchestra.run.smithi043.stdout:55834574893 2024-05-03T10:25:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:24.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:24 smithi043 ceph-mon[22404]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:24.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:24 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2076000328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-03T10:25:24.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:24 smithi043 ceph-mon[27243]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:24.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:24 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2076000328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-03T10:25:24.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:24 smithi181 ceph-mon[23393]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:24.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:24 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2076000328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-03T10:25:26.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:26 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:25:26] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:26.553 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574893 for osd.1 2024-05-03T10:25:26.553 DEBUG:teuthology.parallel:result is None 2024-05-03T10:25:26.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:26 smithi043 ceph-mon[22404]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:26.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:26.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:26.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:26 smithi043 ceph-mon[27243]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:26.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:26.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:26.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:26 smithi181 ceph-mon[23393]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:26.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:26.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:27.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:25:27.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:25:27.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:25:27.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:25:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:25:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:25:28.614 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:28 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:25:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:28 smithi043 ceph-mon[22404]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:29.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:28 smithi043 ceph-mon[27243]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:29.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:28 smithi181 ceph-mon[23393]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:30.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:30 smithi043 ceph-mon[22404]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:30.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:30 smithi043 ceph-mon[27243]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:30 smithi181 ceph-mon[23393]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:31.138 INFO:teuthology.orchestra.run.smithi043.stdout:163208757276 2024-05-03T10:25:31.760 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:31 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/4046657554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-03T10:25:31.760 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:31 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/4046657554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-03T10:25:32.111 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757271 got 163208757276 for osd.5 2024-05-03T10:25:32.111 DEBUG:teuthology.parallel:result is None 2024-05-03T10:25:32.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:31 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/4046657554' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-03T10:25:32.901 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:32 smithi043 ceph-mon[27243]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:32.901 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:32 smithi043 ceph-mon[22404]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:33.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:32 smithi181 ceph-mon[23393]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:33.154 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:33 smithi043 systemd[1]: Started Ceph alertmanager.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:25:33.876 INFO:teuthology.orchestra.run.smithi043.stdout:77309411371 2024-05-03T10:25:34.073 INFO:teuthology.orchestra.run.smithi043.stdout:128849018913 2024-05-03T10:25:34.498 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:34.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:34.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-03T10:25:34.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-03T10:25:34.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: Deploying daemon grafana.a on smithi181 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2189332417' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3488504135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: Deploying daemon grafana.a on smithi181 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2189332417' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-03T10:25:34.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:34 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3488504135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-03T10:25:34.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:34.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-03T10:25:34.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-03T10:25:34.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:34.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: Deploying daemon grafana.a on smithi181 2024-05-03T10:25:34.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2189332417' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-03T10:25:34.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:34 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3488504135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-03T10:25:35.221 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:35 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[46023]: level=info ts=2024-05-03T10:25:35.020Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001034473s 2024-05-03T10:25:35.386 INFO:teuthology.orchestra.run.smithi043.stdout:34359738421 2024-05-03T10:25:36.002 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738421 for osd.0 2024-05-03T10:25:36.002 DEBUG:teuthology.parallel:result is None 2024-05-03T10:25:36.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:36 smithi043 ceph-mon[22404]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:36.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:36 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1340014866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-03T10:25:36.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:36 smithi043 ceph-mon[27243]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:36.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:36 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1340014866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-03T10:25:36.421 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:36 smithi181 ceph-mon[23393]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:36.421 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:36 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1340014866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-03T10:25:36.448 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018909 got 128849018913 for osd.4 2024-05-03T10:25:36.449 DEBUG:teuthology.parallel:result is None 2024-05-03T10:25:37.819 INFO:teuthology.orchestra.run.smithi043.stdout:231928234002 2024-05-03T10:25:38.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:37 smithi043 ceph-mon[22404]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:38.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:37 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:38.265 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:37 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2659867701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-03T10:25:38.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:37 smithi043 ceph-mon[27243]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:38.265 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:37 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:38.266 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:37 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2659867701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-03T10:25:38.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:37 smithi181 ceph-mon[23393]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:38.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:37 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:38.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:37 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2659867701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-03T10:25:38.571 INFO:teuthology.orchestra.run.smithi043.stdout:197568495639 2024-05-03T10:25:38.803 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233999 got 231928234002 for osd.7 2024-05-03T10:25:38.804 DEBUG:teuthology.parallel:result is None 2024-05-03T10:25:39.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:38 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/946834041' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-03T10:25:39.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:38 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/946834041' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-03T10:25:39.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:38 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/946834041' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-03T10:25:39.436 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495639 for osd.6 2024-05-03T10:25:39.436 DEBUG:teuthology.parallel:result is None 2024-05-03T10:25:40.228 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:39 smithi043 ceph-mon[27243]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:40.229 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:39 smithi043 ceph-mon[22404]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:40.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:39 smithi181 ceph-mon[23393]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:40.828 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:40 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:25:40] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:41.198 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411371 for osd.2 2024-05-03T10:25:41.198 DEBUG:teuthology.parallel:result is None 2024-05-03T10:25:41.763 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:41 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:41.763 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:41 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:41.860 INFO:teuthology.orchestra.run.smithi043.stdout:107374182440 2024-05-03T10:25:42.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:41 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:42.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:42 smithi043 ceph-mon[22404]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:42.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:42 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3917612834' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-03T10:25:42.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:42 smithi043 ceph-mon[27243]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:42.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:42 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3917612834' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-03T10:25:42.804 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:42 smithi181 ceph-mon[23393]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:42.805 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:42 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3917612834' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-03T10:25:43.221 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[46023]: level=info ts=2024-05-03T10:25:43.023Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004051544s 2024-05-03T10:25:43.252 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182436 got 107374182440 for osd.3 2024-05-03T10:25:43.252 DEBUG:teuthology.parallel:result is None 2024-05-03T10:25:43.252 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-03T10:25:43.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T10:25:43.799 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:44.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:44 smithi043 ceph-mon[22404]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:44.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:44 smithi043 ceph-mon[27243]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:44.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:44 smithi181 ceph-mon[23393]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:45.486 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:45 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:25:45] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:46.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:46 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:25:46] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:46 smithi181 ceph-mon[23393]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:46.500 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:46 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:46.587 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:46 smithi043 ceph-mon[27243]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:46.587 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:46.587 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:46.587 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:46 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:46.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:46 smithi043 ceph-mon[22404]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:46.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:46.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:46.588 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:46 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:47.023 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:25:47.024 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-05-03T10:25:47.095 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-05-03T10:25:47.095 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-05-03T10:25:47.347 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-05-03T10:25:47.348 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-05-03T10:25:47.349 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-05-03T10:25:47.351 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-05-03T10:25:47.352 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-05-03T10:25:47.353 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-05-03T10:25:47.354 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-05-03T10:25:47.355 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-05-03T10:25:47.356 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-05-03T10:25:47.357 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-05-03T10:25:47.358 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-05-03T10:25:47.359 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-05-03T10:25:47.360 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-05-03T10:25:47.361 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-05-03T10:25:47.362 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-05-03T10:25:47.362 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-05-03T10:25:47.610 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-05-03T10:25:47.610 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-05-03T10:25:47.611 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=500.787667ms 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-05-03T10:25:47.612 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-05-03T10:25:47.613 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-05-03T10:25:47.613 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-05-03T10:25:47.613 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-05-03T10:25:47.613 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-05-03T10:25:47.613 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-05-03T10:25:47.613 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:25:47 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:25:47+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-05-03T10:25:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:25:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:25:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:25:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T10:25:48.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:48.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:25:48.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-03T10:25:48.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: Reconfiguring daemon alertmanager.a on smithi043 2024-05-03T10:25:48.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:48 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:48.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:47 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:25:47] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:48.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:48.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: Reconfiguring daemon alertmanager.a on smithi043 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:48.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:48.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:25:48.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:25:48.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:25:48.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T10:25:48.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:48.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:25:48.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-03T10:25:48.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: Reconfiguring daemon alertmanager.a on smithi043 2024-05-03T10:25:48.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:48 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:48.438 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-05-03T10:25:46.932461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48828,"kb_used_data":6076,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912388,"statfs":{"total":767960285184,"available":767910285312,"internally_reserved":0,"allocated":6221824,"data_stored":3487599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001852"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":267,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-03T10:24:27.003337+0000","last_change":"2024-05-03T10:24:12.614094+0000","last_active":"2024-05-03T10:24:27.003337+0000","last_peered":"2024-05-03T10:24:27.003337+0000","last_clean":"2024-05-03T10:24:27.003337+0000","last_became_active":"2024-05-03T10:24:12.613486+0000","last_became_peered":"2024-05-03T10:24:12.613486+0000","last_unstale":"2024-05-03T10:24:27.003337+0000","last_undegraded":"2024-05-03T10:24:27.003337+0000","last_fullsized":"2024-05-03T10:24:27.003337+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T10:22:07.946098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T10:22:07.946098+0000","last_clean_scrub_stamp":"2024-05-03T10:22:07.946098+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T13:17:25.289821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.55000000000000004}]},{"osd":1,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59899999999999998}]},{"osd":2,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.48499999999999999}]},{"osd":3,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.57099999999999995}]},{"osd":4,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.38800000000000001}]},{"osd":5,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.374},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.42899999999999999}]},{"osd":6,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.40500000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":1101824,"data_stored":765916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.438},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.39100000000000001}]},{"osd":1,"last update":"Fri May 3 10:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri May 3 10:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.379},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.46100000000000002}]},{"osd":3,"last update":"Fri May 3 10:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.42699999999999999}]},{"osd":4,"last update":"Fri May 3 10:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Fri May 3 10:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.41799999999999998}]},{"osd":7,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.45400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.36499999999999999,"15min":0.33100000000000002},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.40899999999999997,"15min":0.377},"min":{"1min":0.35499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri May 3 10:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.39700000000000002,"15min":0.38100000000000001},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.28999999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.56899999999999995,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri May 3 10:25:40 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.432,"5min":0.48499999999999999,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.63400000000000001,"5min":0.754,"15min":0.754},"last":0.63400000000000001}]},{"osd":4,"last update":"Fri May 3 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.79400000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.442,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.69899999999999995}]},{"osd":5,"last update":"Fri May 3 10:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85499999999999998,"5min":1.085,"15min":1.085},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.878,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.60299999999999998}]},{"osd":6,"last update":"Fri May 3 10:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.67800000000000005}]},{"osd":7,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.66800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri May 3 10:24:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.373,"15min":0.35699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.61699999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.39100000000000001,"15min":0.36899999999999999},"min":{"1min":0.318,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.34100000000000003}]},{"osd":2,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63300000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.45500000000000002,"15min":0.46300000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.88400000000000001,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.376}]},{"osd":3,"last update":"Fri May 3 10:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39400000000000002}]},{"osd":4,"last update":"Fri May 3 10:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.60699999999999998,"15min":0.60199999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88700000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri May 3 10:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63900000000000001,"15min":0.63800000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.93500000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.53000000000000003}]},{"osd":6,"last update":"Fri May 3 10:24:46 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.319,"15min":1.319},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":13.768000000000001,"5min":13.768000000000001,"15min":13.768000000000001},"last":0.505},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.495}]},{"osd":7,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.55700000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.27300000000000002,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.443},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41199999999999998,"15min":0.40600000000000003},"min":{"1min":0.26600000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.65400000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri May 3 10:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.29599999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.51600000000000001}]},{"osd":3,"last update":"Fri May 3 10:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47399999999999998,"15min":0.48699999999999999},"min":{"1min":0.30499999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.58399999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.45300000000000001,"15min":0.45900000000000002},"min":{"1min":0.317,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60899999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.502}]},{"osd":4,"last update":"Fri May 3 10:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.92500000000000004,"15min":0.97699999999999998},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81899999999999995,"5min":10.268000000000001,"15min":10.268000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59399999999999997,"15min":0.59699999999999998},"min":{"1min":0.41899999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.66700000000000004}]},{"osd":5,"last update":"Fri May 3 10:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.628,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.745},{"interface":"front","average":{"1min":0.626,"5min":0.995,"15min":1.0569999999999999},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91500000000000004,"5min":10.286,"15min":10.286},"last":0.75600000000000001}]},{"osd":6,"last update":"Fri May 3 10:24:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.76500000000000001}]},{"osd":7,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.77600000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40200000000000002,"15min":0.40999999999999998},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52000000000000002,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.32000000000000001}]},{"osd":1,"last update":"Fri May 3 10:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.438,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66100000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.46100000000000002}]},{"osd":2,"last update":"Fri May 3 10:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47499999999999998,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.64200000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.44600000000000001,"15min":0.45700000000000002},"min":{"1min":0.32900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.59399999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.42499999999999999}]},{"osd":4,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.63,"15min":0.63800000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45100000000000001}]},{"osd":5,"last update":"Fri May 3 10:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.502},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri May 3 10:24:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.57099999999999995}]},{"osd":7,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53700000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.46899999999999997}]},{"osd":1,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.38,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.66100000000000003,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44400000000000001}]},{"osd":2,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.5,"15min":0.499},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71699999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.39500000000000002}]},{"osd":3,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.41199999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69099999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47799999999999998}]},{"osd":5,"last update":"Fri May 3 10:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.379,"15min":0.371},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.36299999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.375}]},{"osd":6,"last update":"Fri May 3 10:24:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.33500000000000002}]},{"osd":7,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.45400000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5688,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739464,"statfs":{"total":95995035648,"available":95989211136,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.52700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69099999999999995,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.749},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.39500000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70199999999999996,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.68100000000000005}]},{"osd":1,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.40400000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79300000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35399999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70999999999999996,"5min":0.752,"15min":0.752},"last":0.79500000000000004}]},{"osd":3,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63800000000000001,"5min":0.752,"15min":0.752},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.748,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61299999999999999}]},{"osd":4,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.71999999999999997,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.42099999999999999,"15min":0.41399999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.46300000000000002}]},{"osd":6,"last update":"Fri May 3 10:24:47 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.439}]},{"osd":7,"last update":"Fri May 3 10:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.498},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T10:25:48.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T10:25:49.284 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:50.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:50 smithi043 ceph-mon[27243]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:50.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:50 smithi043 ceph-mon[22404]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:50.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:50 smithi181 ceph-mon[23393]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:51.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:50 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:25:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:51.542 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:51 smithi043 systemd[1]: Stopping Ceph alertmanager.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:25:52.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:51 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:52.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:51 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:52.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:51 smithi181 ceph-mon[23393]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:52.135 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:51 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:52.135 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:51 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:52.135 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:51 smithi043 ceph-mon[27243]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:52.135 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:51 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:52.135 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:51 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:52.135 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:51 smithi043 ceph-mon[22404]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:52.135 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:52 smithi043 bash[47346]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-03T10:25:53.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:53.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:53.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:25:53] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:53.293 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[46023]: level=info ts=2024-05-03T10:25:53.024Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-03T10:25:53.293 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:53 smithi043 podman[47379]: 2024-05-03 10:25:53.148858956 +0000 UTC m=+1.101376688 container died 1399edded89d41264deb293352f0ad8575a9837d82ddc45cb3b05ebeb9b536c0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-03T10:25:53.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:53.920 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:53 smithi043 podman[47379]: 2024-05-03 10:25:53.558211819 +0000 UTC m=+1.510729543 container cleanup 1399edded89d41264deb293352f0ad8575a9837d82ddc45cb3b05ebeb9b536c0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-03T10:25:53.920 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:53 smithi043 bash[47379]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a 2024-05-03T10:25:53.920 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:53 smithi043 bash[47442]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-03T10:25:54.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:54 smithi043 ceph-mon[22404]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:54.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:54 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:54.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:54 smithi043 ceph-mon[27243]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:54.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:54 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:54.293 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:53 smithi043 podman[47429]: 2024-05-03 10:25:53.918988362 +0000 UTC m=+0.869567904 container remove 1399edded89d41264deb293352f0ad8575a9837d82ddc45cb3b05ebeb9b536c0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-03T10:25:54.293 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:54 smithi043 podman[47429]: 2024-05-03 10:25:54.019919529 +0000 UTC m=+0.970499071 volume remove 1005bbce82e0b4bd672c47bff37dae0256a27b859552516861f3ab87310653fa 2024-05-03T10:25:54.293 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:54 smithi043 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-03T10:25:54.293 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:54 smithi043 systemd[1]: Stopped Ceph alertmanager.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:25:54.293 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:54 smithi043 systemd[1]: Starting Ceph alertmanager.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:25:54.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:54 smithi181 ceph-mon[23393]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:54.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:54 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:55.101 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:54 smithi043 podman[47537]: 2024-05-03 10:25:54.888279856 +0000 UTC m=+0.024859595 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-05-03T10:25:55.101 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 podman[47537]: 2024-05-03 10:25:55.000055512 +0000 UTC m=+0.136635255 volume create e13995d97c8c0179009c9530866b8943c9f223d444278d02f87925d681b9b63f 2024-05-03T10:25:55.102 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:25:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-05-03T10:25:55.542 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 podman[47537]: 2024-05-03 10:25:55.108950098 +0000 UTC m=+0.245529831 container create 43fbcd25b859fb929b75218655c4ebc7df1235a5e12906546f0ef186c3e111ad (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 podman[47537]: 2024-05-03 10:25:55.629941208 +0000 UTC m=+0.766520941 container init 43fbcd25b859fb929b75218655c4ebc7df1235a5e12906546f0ef186c3e111ad (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 podman[47537]: 2024-05-03 10:25:55.633090583 +0000 UTC m=+0.769670317 container start 43fbcd25b859fb929b75218655c4ebc7df1235a5e12906546f0ef186c3e111ad (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:25:55.644Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:25:55.644Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:25:55.645Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.43 port=9094 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:25:55.647Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:25:55.672Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:25:55.673Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:25:55.675Z caller=main.go:518 msg=Listening address=:9093 2024-05-03T10:25:55.876 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:25:55.675Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-05-03T10:25:55.877 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 bash[47537]: 43fbcd25b859fb929b75218655c4ebc7df1235a5e12906546f0ef186c3e111ad 2024-05-03T10:25:55.877 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:55 smithi043 systemd[1]: Started Ceph alertmanager.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:25:56.276 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:56 smithi043 ceph-mon[22404]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:56.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:56 smithi043 ceph-mon[22404]: from='client.14607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T10:25:56.277 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:56 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:56.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:56 smithi043 ceph-mon[27243]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:56.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:56 smithi043 ceph-mon[27243]: from='client.14607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T10:25:56.277 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:56 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:56.301 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-05-03T10:25:54.933528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48828,"kb_used_data":6076,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912388,"statfs":{"total":767960285184,"available":767910285312,"internally_reserved":0,"allocated":6221824,"data_stored":3487599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001696"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":267,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-05-03T10:24:27.003337+0000","last_change":"2024-05-03T10:24:12.614094+0000","last_active":"2024-05-03T10:24:27.003337+0000","last_peered":"2024-05-03T10:24:27.003337+0000","last_clean":"2024-05-03T10:24:27.003337+0000","last_became_active":"2024-05-03T10:24:12.613486+0000","last_became_peered":"2024-05-03T10:24:12.613486+0000","last_unstale":"2024-05-03T10:24:27.003337+0000","last_undegraded":"2024-05-03T10:24:27.003337+0000","last_fullsized":"2024-05-03T10:24:27.003337+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T10:22:07.946098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T10:22:07.946098+0000","last_clean_scrub_stamp":"2024-05-03T10:22:07.946098+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T13:17:25.289821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51800000000000002}]},{"osd":2,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.48499999999999999}]},{"osd":4,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.35699999999999998}]},{"osd":5,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.41799999999999998}]},{"osd":6,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.438}]}]},{"osd":6,"up_from":46,"seq":197568495643,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":1101824,"data_stored":765916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.48699999999999999}]},{"osd":1,"last update":"Fri May 3 10:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.502,"15min":0.5},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.503,"15min":0.495},"min":{"1min":0.41799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri May 3 10:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.499,"15min":0.49399999999999999},"min":{"1min":0.379,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.51800000000000002,"15min":0.51300000000000001},"min":{"1min":0.434,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.52300000000000002}]},{"osd":3,"last update":"Fri May 3 10:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53100000000000003,"15min":0.52300000000000002},"min":{"1min":0.38600000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.54700000000000004}]},{"osd":4,"last update":"Fri May 3 10:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.42599999999999999,"15min":0.41599999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.439,"15min":0.42699999999999999},"min":{"1min":0.36499999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.434}]},{"osd":5,"last update":"Fri May 3 10:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.45300000000000001,"15min":0.441},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.44400000000000001,"15min":0.43099999999999999},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69799999999999995,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.44700000000000001}]},{"osd":7,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.38100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38200000000000001,"15min":0.33700000000000002},"min":{"1min":0.28199999999999997,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41899999999999998,"15min":0.38100000000000001},"min":{"1min":0.30299999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61899999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42099999999999999}]},{"osd":2,"last update":"Fri May 3 10:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.39700000000000002,"15min":0.38100000000000001},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.28999999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.56899999999999995,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.33600000000000002}]},{"osd":3,"last update":"Fri May 3 10:25:40 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.48499999999999999,"15min":0.48699999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.63400000000000001,"5min":0.754,"15min":0.754},"last":0.35099999999999998}]},{"osd":4,"last update":"Fri May 3 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.79400000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.442,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri May 3 10:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85499999999999998,"5min":1.085,"15min":1.085},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.878,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.435}]},{"osd":6,"last update":"Fri May 3 10:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.63700000000000001,"15min":0.64700000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.76300000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.69599999999999995,"15min":0.70299999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.876,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.59699999999999998}]},{"osd":7,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.497},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri May 3 10:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.35999999999999999},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.40300000000000002,"15min":0.373},"min":{"1min":0.28299999999999997,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63300000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.45500000000000002,"15min":0.46300000000000002},"min":{"1min":0.29899999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.88400000000000001,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri May 3 10:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.35099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.436,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.52000000000000002}]},{"osd":4,"last update":"Fri May 3 10:25:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.60699999999999998,"15min":0.60199999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88700000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.79200000000000004}]},{"osd":5,"last update":"Fri May 3 10:25:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.63900000000000001,"15min":0.63800000000000001},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.93500000000000005,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.64200000000000002}]},{"osd":6,"last update":"Fri May 3 10:25:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.173,"15min":1.27},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.874,"5min":13.768000000000001,"15min":13.768000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.64400000000000002,"15min":0.64800000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.72499999999999998}]},{"osd":7,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40500000000000003},"min":{"1min":0.27300000000000002,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41199999999999998,"15min":0.40600000000000003},"min":{"1min":0.26600000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.65400000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.29399999999999998}]},{"osd":1,"last update":"Fri May 3 10:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.29599999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.307},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.377}]},{"osd":3,"last update":"Fri May 3 10:25:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47399999999999998,"15min":0.48699999999999999},"min":{"1min":0.30499999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.58399999999999996,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.45300000000000001,"15min":0.45900000000000002},"min":{"1min":0.317,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60899999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.32400000000000001}]},{"osd":4,"last update":"Fri May 3 10:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.92500000000000004,"15min":0.97699999999999998},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81899999999999995,"5min":10.268000000000001,"15min":10.268000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59399999999999997,"15min":0.59699999999999998},"min":{"1min":0.41899999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.624}]},{"osd":5,"last update":"Fri May 3 10:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.628,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.995,"15min":1.0569999999999999},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91500000000000004,"5min":10.286,"15min":10.286},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri May 3 10:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63100000000000001,"15min":0.63},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.69299999999999995}]},{"osd":7,"last update":"Fri May 3 10:25:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76900000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.40200000000000002,"15min":0.40999999999999998},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.52000000000000002,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.39100000000000001}]},{"osd":1,"last update":"Fri May 3 10:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.307},{"interface":"front","average":{"1min":0.438,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66100000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri May 3 10:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47499999999999998,"15min":0.48099999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.64200000000000002,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.37},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.44600000000000001,"15min":0.45700000000000002},"min":{"1min":0.32900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.59399999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.374}]},{"osd":4,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.63,"15min":0.63800000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.61699999999999999}]},{"osd":5,"last update":"Fri May 3 10:25:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63800000000000001,"15min":0.64200000000000002},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.77100000000000002}]},{"osd":6,"last update":"Fri May 3 10:25:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.76000000000000001}]},{"osd":7,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65200000000000002}]}]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.52600000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.54900000000000004}]},{"osd":1,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.38,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.66100000000000003,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.629}]},{"osd":2,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.5,"15min":0.499},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.71699999999999997,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri May 3 10:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.41199999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69099999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri May 3 10:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.379,"15min":0.371},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.36299999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.48599999999999999}]},{"osd":6,"last update":"Fri May 3 10:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40300000000000002,"15min":0.39600000000000002},"min":{"1min":0.307,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.42699999999999999,"15min":0.41499999999999998},"min":{"1min":0.33500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.38200000000000001}]},{"osd":7,"last update":"Fri May 3 10:25:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.498}]}]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5688,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739464,"statfs":{"total":95995035648,"available":95989211136,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.52700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69099999999999995,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.39500000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70199999999999996,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.54700000000000004}]},{"osd":1,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.40400000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79300000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.501}]},{"osd":2,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35399999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70999999999999996,"5min":0.752,"15min":0.752},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.63800000000000001,"5min":0.752,"15min":0.752},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.748,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri May 3 10:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.71999999999999997,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.378},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.42099999999999999,"15min":0.41399999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45400000000000001}]},{"osd":6,"last update":"Fri May 3 10:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.44,"15min":0.44},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.61799999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.311,"5min":0.31,"15min":0.31},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.437}]},{"osd":7,"last update":"Fri May 3 10:25:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T10:25:56.302 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-03T10:25:56.302 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-03T10:25:56.302 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-03T10:25:56.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -- ceph health --format=json 2024-05-03T10:25:56.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:56 smithi181 ceph-mon[23393]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:56.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:56 smithi181 ceph-mon[23393]: from='client.14607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T10:25:56.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:56 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:Inferring config /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T10:25:57.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:57 smithi043 ceph-mon[22404]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-03T10:25:57.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:57 smithi043 ceph-mon[22404]: Reconfiguring daemon prometheus.a on smithi181 2024-05-03T10:25:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:57 smithi043 ceph-mon[27243]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-03T10:25:57.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:57 smithi043 ceph-mon[27243]: Reconfiguring daemon prometheus.a on smithi181 2024-05-03T10:25:57.794 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:25:57 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:25:57.647Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000400088s 2024-05-03T10:25:58.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:57 smithi181 ceph-mon[23393]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-03T10:25:58.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:57 smithi181 ceph-mon[23393]: Reconfiguring daemon prometheus.a on smithi181 2024-05-03T10:25:58.744 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:58 smithi043 ceph-mon[22404]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:58.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:58 smithi043 ceph-mon[27243]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:58.745 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:58 smithi181 ceph-mon[23393]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:25:59.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:25:58 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:25:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:25:59.739 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:59 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:25:59.740 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:59 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:00.004 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:26:00.004 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-03T10:26:00.004 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:59 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:00.004 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:25:59 smithi043 ceph-mon[27243]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:00.004 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:59 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:00.005 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:25:59 smithi043 ceph-mon[22404]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:00.063 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:59 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:00.063 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:59 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:00.063 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:25:59 smithi181 ceph-mon[23393]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:00.684 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-03T10:26:00.685 INFO:tasks.cephadm:Setup complete, yielding 2024-05-03T10:26:00.685 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-03T10:26:00.695 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi043.front.sepia.ceph.com 2024-05-03T10:26:00.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-05-03T10:26:00.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:00 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/205261183' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-03T10:26:01.037 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:00 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/205261183' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-03T10:26:01.037 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:00 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/205261183' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-03T10:26:01.619 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 systemd[1]: Stopping Ceph prometheus.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 bash[42759]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.474Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.474Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.474Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.474Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.474Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.475Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.475Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.475Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.475Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.475Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.475Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[39831]: ts=2024-05-03T10:26:01.475Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-03T10:26:01.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 podman[42772]: 2024-05-03 10:26:01.485389963 +0000 UTC m=+0.119091341 container died 306833a359290518403be9c77ba9e00a9ac1d489fa98564ed0ac80d4bb958aef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:26:02.110 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 podman[42772]: 2024-05-03 10:26:01.849640575 +0000 UTC m=+0.483341992 container cleanup 306833a359290518403be9c77ba9e00a9ac1d489fa98564ed0ac80d4bb958aef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:26:02.110 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 bash[42772]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a 2024-05-03T10:26:02.110 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:01 smithi181 bash[42798]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-03T10:26:02.360 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 podman[42786]: 2024-05-03 10:26:02.108979847 +0000 UTC m=+0.623062222 container remove 306833a359290518403be9c77ba9e00a9ac1d489fa98564ed0ac80d4bb958aef (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:26:02.361 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-03T10:26:02.361 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 systemd[1]: Stopped Ceph prometheus.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:26:02.361 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 systemd[1]: Starting Ceph prometheus.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:26:02.361 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 podman[42867]: 2024-05-03 10:26:02.25106795 +0000 UTC m=+0.018876617 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-03T10:26:02.361 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 podman[42867]: 2024-05-03 10:26:02.359846024 +0000 UTC m=+0.127654691 container create 906505cdab90f76f01c973b91b54cfa7941cb388f0454833584529a2c6d313cd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:26:02.619 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 podman[42867]: 2024-05-03 10:26:02.55374199 +0000 UTC m=+0.321550657 container init 906505cdab90f76f01c973b91b54cfa7941cb388f0454833584529a2c6d313cd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 podman[42867]: 2024-05-03 10:26:02.556383019 +0000 UTC m=+0.324191687 container start 906505cdab90f76f01c973b91b54cfa7941cb388f0454833584529a2c6d313cd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.579Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.579Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.579Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.579Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi181 (none))" 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.579Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.579Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.581Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.581Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.582Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.584Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-03T10:26:02.620 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.584Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.056µs 2024-05-03T10:26:02.621 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.584Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-03T10:26:02.621 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.585Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-05-03T10:26:02.621 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.585Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-05-03T10:26:02.621 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.585Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=46.733µs wal_replay_duration=801.183µs total_replay_duration=916.548µs 2024-05-03T10:26:02.621 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.586Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-03T10:26:02.621 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.586Z caller=main.go:947 level=info msg="TSDB started" 2024-05-03T10:26:02.621 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.586Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-03T10:26:02.621 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.601Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.675221ms db_storage=700ns remote_storage=1.109µs web_handler=271ns query_engine=697ns scrape=292.255µs scrape_sd=38.39µs notify=17.996µs notify_sd=5.386µs rules=14.107304ms 2024-05-03T10:26:02.621 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:26:02.601Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-03T10:26:02.998 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 bash[42867]: 906505cdab90f76f01c973b91b54cfa7941cb388f0454833584529a2c6d313cd 2024-05-03T10:26:02.998 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:26:02 smithi181 systemd[1]: Started Ceph prometheus.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:26:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:02 smithi043 ceph-mon[22404]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:03.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:02 smithi043 ceph-mon[27243]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:03.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:02 smithi181 ceph-mon[23393]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:04.091 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:04.091 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.43:9093"}]: dispatch 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.43:9093"}]: dispatch 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.181:3000"}]: dispatch 2024-05-03T10:26:04.092 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.181:3000"}]: dispatch 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: osdmap e60: 8 total, 8 up, 8 in 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1855722242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-03T10:26:04.093 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.43:9093"}]: dispatch 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.43:9093"}]: dispatch 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-03T10:26:04.094 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.181:3000"}]: dispatch 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.181:3000"}]: dispatch 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: osdmap e60: 8 total, 8 up, 8 in 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1855722242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-03T10:26:04.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:04 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-03T10:26:04.096 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-03T10:26:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b22db950-3b56-45cd-a14b-b934aaa70506", 2024-05-03T10:26:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "r", 2024-05-03T10:26:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "current_period": "a16d84d5-b987-4196-9192-9a9008e60dde", 2024-05-03T10:26:04.096 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1 2024-05-03T10:26:04.096 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-03T10:26:04.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:04.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-03T10:26:04.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-03T10:26:04.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.43:9093"}]: dispatch 2024-05-03T10:26:04.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.43:9093"}]: dispatch 2024-05-03T10:26:04.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:26:04.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-03T10:26:04.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.181:3000"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.181:3000"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: osdmap e60: 8 total, 8 up, 8 in 2024-05-03T10:26:04.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1855722242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-03T10:26:04.372 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:04 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-03T10:26:05.108 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:05 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-03T10:26:05.109 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:05 smithi043 ceph-mon[22404]: osdmap e61: 8 total, 8 up, 8 in 2024-05-03T10:26:05.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:05 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-03T10:26:05.109 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:05 smithi043 ceph-mon[27243]: osdmap e61: 8 total, 8 up, 8 in 2024-05-03T10:26:05.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:05 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-03T10:26:05.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:05 smithi181 ceph-mon[23393]: osdmap e61: 8 total, 8 up, 8 in 2024-05-03T10:26:05.760 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:26:05 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:26:05.649Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002323127s 2024-05-03T10:26:05.788 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-05-03T10:26:06.351 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:06 smithi043 ceph-mon[22404]: pgmap v54: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:06.352 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:06 smithi043 ceph-mon[22404]: osdmap e62: 8 total, 8 up, 8 in 2024-05-03T10:26:06.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:06 smithi043 ceph-mon[27243]: pgmap v54: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:06.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:06 smithi043 ceph-mon[27243]: osdmap e62: 8 total, 8 up, 8 in 2024-05-03T10:26:06.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:06 smithi181 ceph-mon[23393]: pgmap v54: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:06.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:06 smithi181 ceph-mon[23393]: osdmap e62: 8 total, 8 up, 8 in 2024-05-03T10:26:06.740 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:06 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:26:06] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:07.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:07 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:07.597 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:07 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:07.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:07 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:07.598 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:07 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:07.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:07 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:07.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:07 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:07.998 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:07 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:26:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:08.409 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "2e4dfa64-3f0b-471d-aa96-66913a71c1a9", 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default", 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "default", 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": "true", 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [], 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "", 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [], 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [], 2024-05-03T10:26:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "", 2024-05-03T10:26:08.411 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "b22db950-3b56-45cd-a14b-b934aaa70506", 2024-05-03T10:26:08.411 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-05-03T10:26:08.411 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-05-03T10:26:08.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:08.411 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-03T10:26:08.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:08 smithi181 ceph-mon[23393]: pgmap v56: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:08.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:08 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:08.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:08 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:09.039 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:08 smithi043 ceph-mon[27243]: pgmap v56: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:09.039 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:08 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:09.039 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:08 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:09.039 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:08 smithi043 ceph-mon[22404]: pgmap v56: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:09.040 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:08 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:09.040 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:08 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:09.068 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-05-03T10:26:09.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:26:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:26:10.640 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:10 smithi043 ceph-mon[27243]: pgmap v57: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-03T10:26:10.640 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:10 smithi043 ceph-mon[22404]: pgmap v57: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-03T10:26:10.824 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-03T10:26:10.824 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "992bb9c3-46cd-49b1-9cc9-580307298d0d", 2024-05-03T10:26:10.824 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "z", 2024-05-03T10:26:10.824 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "z.rgw.meta:root", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "z.rgw.control", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "z.rgw.log:gc", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "z.rgw.log:lc", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "z.rgw.log", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "z.rgw.meta:roles", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-05-03T10:26:10.825 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "z.rgw.otp", 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "", 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "" 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "z.rgw.buckets.index", 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "z.rgw.buckets.data" 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:26:10.826 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-05-03T10:26:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0 2024-05-03T10:26:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-03T10:26:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "b22db950-3b56-45cd-a14b-b934aaa70506", 2024-05-03T10:26:10.827 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "z.rgw.log:notif" 2024-05-03T10:26:10.827 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-03T10:26:10.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:10 smithi181 ceph-mon[23393]: pgmap v57: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-05-03T10:26:11.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-05-03T10:26:12.721 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:12 smithi181 ceph-mon[23393]: pgmap v58: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s wr, 1 op/s 2024-05-03T10:26:12.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:12 smithi043 ceph-mon[22404]: pgmap v58: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s wr, 1 op/s 2024-05-03T10:26:12.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:12 smithi043 ceph-mon[27243]: pgmap v58: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s wr, 1 op/s 2024-05-03T10:26:13.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:13] "GET /metrics HTTP/1.1" 200 192905 "" "Prometheus/2.33.4" 2024-05-03T10:26:14.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:13 smithi181 ceph-mon[23393]: pgmap v59: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 806 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-03T10:26:14.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:13 smithi043 ceph-mon[22404]: pgmap v59: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 806 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-03T10:26:14.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:13 smithi043 ceph-mon[27243]: pgmap v59: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 806 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-05-03T10:26:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:14 smithi181 ceph-mon[23393]: osdmap e63: 8 total, 8 up, 8 in 2024-05-03T10:26:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:14 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-03T10:26:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:14 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-03T10:26:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:14 smithi181 ceph-mon[23393]: osdmap e64: 8 total, 8 up, 8 in 2024-05-03T10:26:15.193 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:14 smithi043 ceph-mon[27243]: osdmap e63: 8 total, 8 up, 8 in 2024-05-03T10:26:15.194 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:14 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-03T10:26:15.194 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:14 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-03T10:26:15.194 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:14 smithi043 ceph-mon[27243]: osdmap e64: 8 total, 8 up, 8 in 2024-05-03T10:26:15.194 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:14 smithi043 ceph-mon[22404]: osdmap e63: 8 total, 8 up, 8 in 2024-05-03T10:26:15.194 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:14 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-03T10:26:15.194 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:14 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-03T10:26:15.194 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:14 smithi043 ceph-mon[22404]: osdmap e64: 8 total, 8 up, 8 in 2024-05-03T10:26:16.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:15 smithi043 ceph-mon[22404]: pgmap v62: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-05-03T10:26:16.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:15 smithi043 ceph-mon[22404]: osdmap e65: 8 total, 8 up, 8 in 2024-05-03T10:26:16.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:15 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-03T10:26:16.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:15 smithi043 ceph-mon[27243]: pgmap v62: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-05-03T10:26:16.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:15 smithi043 ceph-mon[27243]: osdmap e65: 8 total, 8 up, 8 in 2024-05-03T10:26:16.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:15 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-03T10:26:16.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:15 smithi181 ceph-mon[23393]: pgmap v62: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-05-03T10:26:16.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:15 smithi181 ceph-mon[23393]: osdmap e65: 8 total, 8 up, 8 in 2024-05-03T10:26:16.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:15 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-03T10:26:17.911 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:17 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-03T10:26:17.911 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:17 smithi181 ceph-mon[23393]: osdmap e66: 8 total, 8 up, 8 in 2024-05-03T10:26:17.911 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:17 smithi181 ceph-mon[23393]: pgmap v65: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:18.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:17 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-03T10:26:18.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:17 smithi043 ceph-mon[22404]: osdmap e66: 8 total, 8 up, 8 in 2024-05-03T10:26:18.133 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:17 smithi043 ceph-mon[22404]: pgmap v65: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:18.133 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:17 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3515215246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-03T10:26:18.134 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:17 smithi043 ceph-mon[27243]: osdmap e66: 8 total, 8 up, 8 in 2024-05-03T10:26:18.134 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:17 smithi043 ceph-mon[27243]: pgmap v65: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:18.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:18 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:26:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:18 smithi181 ceph-mon[23393]: osdmap e67: 8 total, 8 up, 8 in 2024-05-03T10:26:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:18 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3587143813' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-03T10:26:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:18 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-03T10:26:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:18 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:18 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:19.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[22404]: osdmap e67: 8 total, 8 up, 8 in 2024-05-03T10:26:19.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3587143813' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-03T10:26:19.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-03T10:26:19.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:19.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:19.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[27243]: osdmap e67: 8 total, 8 up, 8 in 2024-05-03T10:26:19.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3587143813' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-03T10:26:19.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-03T10:26:19.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:19.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:18 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:19.824 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:26:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:26:20.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:19 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-03T10:26:20.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:19 smithi181 ceph-mon[23393]: osdmap e68: 8 total, 8 up, 8 in 2024-05-03T10:26:20.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:19 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3587143813' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-03T10:26:20.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:19 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-03T10:26:20.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:19 smithi181 ceph-mon[23393]: pgmap v68: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T10:26:20.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-03T10:26:20.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[27243]: osdmap e68: 8 total, 8 up, 8 in 2024-05-03T10:26:20.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3587143813' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-03T10:26:20.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-03T10:26:20.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[27243]: pgmap v68: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T10:26:20.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-03T10:26:20.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[22404]: osdmap e68: 8 total, 8 up, 8 in 2024-05-03T10:26:20.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3587143813' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-03T10:26:20.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-03T10:26:20.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:19 smithi043 ceph-mon[22404]: pgmap v68: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T10:26:20.888 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-03T10:26:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "7a9d9f8b-63d4-4690-9bcd-f1fd81c7891a", 2024-05-03T10:26:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-05-03T10:26:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "a16d84d5-b987-4196-9192-9a9008e60dde", 2024-05-03T10:26:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "7a9d9f8b-63d4-4690-9bcd-f1fd81c7891a", 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "2e4dfa64-3f0b-471d-aa96-66913a71c1a9", 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default", 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "default", 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": "true", 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [], 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-05-03T10:26:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "992bb9c3-46cd-49b1-9cc9-580307298d0d", 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "992bb9c3-46cd-49b1-9cc9-580307298d0d", 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "z", 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [], 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": "false", 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": "false", 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": "false", 2024-05-03T10:26:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": "true", 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "" 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-03T10:26:20.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "b22db950-3b56-45cd-a14b-b934aaa70506", 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "992bb9c3-46cd-49b1-9cc9-580307298d0d", 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 218538463 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:26:20.892 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "2e4dfa64-3f0b-471d-aa96-66913a71c1a9", 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "992bb9c3-46cd-49b1-9cc9-580307298d0d", 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-05-03T10:26:20.893 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-05-03T10:26:20.894 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "b22db950-3b56-45cd-a14b-b934aaa70506", 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "r", 2024-05-03T10:26:20.895 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-05-03T10:26:20.896 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-03T10:26:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:20 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-03T10:26:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:20 smithi181 ceph-mon[23393]: osdmap e69: 8 total, 8 up, 8 in 2024-05-03T10:26:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:20 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3587143813' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-03T10:26:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:20 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-03T10:26:21.185 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:20 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-03T10:26:21.185 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:20 smithi043 ceph-mon[27243]: osdmap e69: 8 total, 8 up, 8 in 2024-05-03T10:26:21.185 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:20 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3587143813' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-03T10:26:21.185 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:20 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-03T10:26:21.185 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:20 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-03T10:26:21.185 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:20 smithi043 ceph-mon[22404]: osdmap e69: 8 total, 8 up, 8 in 2024-05-03T10:26:21.185 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:20 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3587143813' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-03T10:26:21.186 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:20 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-03T10:26:21.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-05-03T10:26:21.878 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:21 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-03T10:26:21.878 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:21 smithi043 ceph-mon[22404]: osdmap e70: 8 total, 8 up, 8 in 2024-05-03T10:26:21.878 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:21 smithi043 ceph-mon[22404]: pgmap v71: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T10:26:21.879 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:21 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-03T10:26:21.879 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:21 smithi043 ceph-mon[27243]: osdmap e70: 8 total, 8 up, 8 in 2024-05-03T10:26:21.879 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:21 smithi043 ceph-mon[27243]: pgmap v71: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T10:26:22.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:21 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-03T10:26:22.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:21 smithi181 ceph-mon[23393]: osdmap e70: 8 total, 8 up, 8 in 2024-05-03T10:26:22.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:21 smithi181 ceph-mon[23393]: pgmap v71: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T10:26:23.462 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:23] "GET /metrics HTTP/1.1" 200 192905 "" "Prometheus/2.33.4" 2024-05-03T10:26:24.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:23 smithi043 ceph-mon[22404]: pgmap v72: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-05-03T10:26:24.046 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:23 smithi043 ceph-mon[27243]: pgmap v72: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-05-03T10:26:24.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:23 smithi181 ceph-mon[23393]: pgmap v72: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-05-03T10:26:25.746 INFO:teuthology.orchestra.run.smithi043.stdout:Scheduled rgw.foo update... 2024-05-03T10:26:26.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph osd pool create foo' 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: pgmap v73: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: Saving service rgw.foo spec with placement count:2 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.745 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: Saving service rgw.foo spec with placement count:2 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi043.vtftcc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi043.vtftcc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[27243]: Deploying daemon rgw.foo.smithi043.vtftcc on smithi043 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: pgmap v73: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: Saving service rgw.foo spec with placement count:2 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.746 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: Saving service rgw.foo spec with placement count:2 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi043.vtftcc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi043.vtftcc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:26.747 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:26 smithi043 ceph-mon[22404]: Deploying daemon rgw.foo.smithi043.vtftcc on smithi043 2024-05-03T10:26:26.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: pgmap v73: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-03T10:26:26.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:26:26.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: Saving service rgw.foo spec with placement count:2 2024-05-03T10:26:26.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:26.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:26.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: Saving service rgw.foo spec with placement count:2 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi043.vtftcc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi043.vtftcc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:26.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:26 smithi181 ceph-mon[23393]: Deploying daemon rgw.foo.smithi043.vtftcc on smithi043 2024-05-03T10:26:27.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-05-03T10:26:27.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-05-03T10:26:27.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-05-03T10:26:27.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-05-03T10:26:27.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:26:27.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:26:27.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-05-03T10:26:27.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-05-03T10:26:27.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-05-03T10:26:27.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-05-03T10:26:27.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:26:27.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:26:28.110 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:27 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:26:27] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-05-03T10:26:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-05-03T10:26:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-05-03T10:26:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-05-03T10:26:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:26:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:26:28.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[22404]: pgmap v74: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-05-03T10:26:28.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-05-03T10:26:28.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-05-03T10:26:28.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-05-03T10:26:28.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-05-03T10:26:28.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[22404]: osdmap e71: 8 total, 8 up, 8 in 2024-05-03T10:26:28.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:28.932 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:28.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[27243]: pgmap v74: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-05-03T10:26:28.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-05-03T10:26:28.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-05-03T10:26:28.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-05-03T10:26:28.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-05-03T10:26:28.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[27243]: osdmap e71: 8 total, 8 up, 8 in 2024-05-03T10:26:28.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:28.933 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:28 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:28 smithi181 ceph-mon[23393]: pgmap v74: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-05-03T10:26:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-05-03T10:26:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-05-03T10:26:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-05-03T10:26:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-05-03T10:26:29.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:28 smithi181 ceph-mon[23393]: osdmap e71: 8 total, 8 up, 8 in 2024-05-03T10:26:29.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:29.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:28 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:29.852 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:26:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:26:30.095 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:30 smithi043 ceph-mon[27243]: osdmap e72: 8 total, 8 up, 8 in 2024-05-03T10:26:30.096 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:30 smithi043 ceph-mon[22404]: osdmap e72: 8 total, 8 up, 8 in 2024-05-03T10:26:30.096 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'foo' created 2024-05-03T10:26:30.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:30 smithi181 ceph-mon[23393]: osdmap e72: 8 total, 8 up, 8 in 2024-05-03T10:26:30.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'rbd pool init foo' 2024-05-03T10:26:31.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: pgmap v77: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-05-03T10:26:31.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2698605912' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-03T10:26:31.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-03T10:26:31.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:31.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi181.kgwjoi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-03T10:26:31.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi181.kgwjoi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-03T10:26:31.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:31.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:31.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: Deploying daemon rgw.foo.smithi181.kgwjoi on smithi181 2024-05-03T10:26:31.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-03T10:26:31.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:31 smithi181 ceph-mon[23393]: osdmap e73: 8 total, 8 up, 8 in 2024-05-03T10:26:31.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: pgmap v77: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-05-03T10:26:31.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2698605912' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-03T10:26:31.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-03T10:26:31.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:31.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi181.kgwjoi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi181.kgwjoi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: Deploying daemon rgw.foo.smithi181.kgwjoi on smithi181 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[22404]: osdmap e73: 8 total, 8 up, 8 in 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: pgmap v77: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 895 B/s wr, 2 op/s 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2698605912' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi181.kgwjoi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-03T10:26:31.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi181.kgwjoi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-03T10:26:31.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:31.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:31.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: Deploying daemon rgw.foo.smithi181.kgwjoi on smithi181 2024-05-03T10:26:31.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-03T10:26:31.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:31 smithi043 ceph-mon[27243]: osdmap e73: 8 total, 8 up, 8 in 2024-05-03T10:26:32.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:32 smithi181 ceph-mon[23393]: pgmap v79: 161 pgs: 32 unknown, 4 peering, 125 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:32.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:32 smithi181 ceph-mon[23393]: osdmap e74: 8 total, 8 up, 8 in 2024-05-03T10:26:32.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:32 smithi043 ceph-mon[22404]: pgmap v79: 161 pgs: 32 unknown, 4 peering, 125 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:32.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:32 smithi043 ceph-mon[22404]: osdmap e74: 8 total, 8 up, 8 in 2024-05-03T10:26:32.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:32 smithi043 ceph-mon[27243]: pgmap v79: 161 pgs: 32 unknown, 4 peering, 125 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:32.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:32 smithi043 ceph-mon[27243]: osdmap e74: 8 total, 8 up, 8 in 2024-05-03T10:26:33.330 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:33 smithi181 ceph-mon[23393]: osdmap e75: 8 total, 8 up, 8 in 2024-05-03T10:26:33.331 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:33 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/4265637475' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-03T10:26:33.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:33 smithi043 ceph-mon[27243]: osdmap e75: 8 total, 8 up, 8 in 2024-05-03T10:26:33.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:33 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/4265637475' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-03T10:26:33.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:33 smithi043 ceph-mon[22404]: osdmap e75: 8 total, 8 up, 8 in 2024-05-03T10:26:33.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:33 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/4265637475' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-03T10:26:33.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:33 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:33] "GET /metrics HTTP/1.1" 200 196732 "" "Prometheus/2.33.4" 2024-05-03T10:26:34.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: pgmap v82: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 7.7 KiB/s wr, 260 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T10:26:34.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/4265637475' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-03T10:26:34.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: osdmap e76: 8 total, 8 up, 8 in 2024-05-03T10:26:34.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:34.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: Checking dashboard <-> RGW credentials 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: Checking dashboard <-> RGW credentials 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:34 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:34.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: pgmap v82: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 7.7 KiB/s wr, 260 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T10:26:34.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/4265637475' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-03T10:26:34.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: osdmap e76: 8 total, 8 up, 8 in 2024-05-03T10:26:34.448 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: Checking dashboard <-> RGW credentials 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: Checking dashboard <-> RGW credentials 2024-05-03T10:26:34.449 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: pgmap v82: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 7.7 KiB/s wr, 260 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/4265637475' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: osdmap e76: 8 total, 8 up, 8 in 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: Checking dashboard <-> RGW credentials 2024-05-03T10:26:34.450 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.451 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.451 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.451 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:34.451 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:34.451 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:34.451 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:34.451 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: Checking dashboard <-> RGW credentials 2024-05-03T10:26:34.451 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:34.451 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:34 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:35.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:35 smithi181 ceph-mon[23393]: osdmap e77: 8 total, 8 up, 8 in 2024-05-03T10:26:35.384 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:35 smithi043 ceph-mon[22404]: osdmap e77: 8 total, 8 up, 8 in 2024-05-03T10:26:35.384 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:35 smithi043 ceph-mon[27243]: osdmap e77: 8 total, 8 up, 8 in 2024-05-03T10:26:36.337 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph orch apply iscsi foo u p' 2024-05-03T10:26:36.362 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:36 smithi043 ceph-mon[27243]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 8.2 KiB/s wr, 280 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T10:26:36.363 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:36 smithi043 ceph-mon[27243]: osdmap e78: 8 total, 8 up, 8 in 2024-05-03T10:26:36.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:36 smithi043 ceph-mon[22404]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 8.2 KiB/s wr, 280 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T10:26:36.363 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:36 smithi043 ceph-mon[22404]: osdmap e78: 8 total, 8 up, 8 in 2024-05-03T10:26:36.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:36 smithi181 ceph-mon[23393]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 8.2 KiB/s wr, 280 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T10:26:36.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:36 smithi181 ceph-mon[23393]: osdmap e78: 8 total, 8 up, 8 in 2024-05-03T10:26:37.198 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:36 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:26:36] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:38.221 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:37 smithi043 ceph-mon[22404]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 6.8 KiB/s wr, 231 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T10:26:38.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:37 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:38.222 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:37 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:38.222 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:37 smithi043 ceph-mon[27243]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 6.8 KiB/s wr, 231 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T10:26:38.222 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:37 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:38.222 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:37 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:38.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:37 smithi181 ceph-mon[23393]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 6.8 KiB/s wr, 231 op/s; 0 B/s, 0 objects/s recovering 2024-05-03T10:26:38.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:37 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:38.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:37 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:38.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:26:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:38.924 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:38 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:26:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:39.363 INFO:teuthology.orchestra.run.smithi043.stdout:Scheduled iscsi.foo update... 2024-05-03T10:26:39.481 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:26:39 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:26:39.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:39 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:39.481 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:39 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:39.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:39 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:39.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:39 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:39.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:39 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:39.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:39 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:40 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:26:40] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:40.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 853 B/s wr, 180 op/s 2024-05-03T10:26:40.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: from='client.24697 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: Saving service iscsi.foo spec with placement count:1 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.740 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:40 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:26:40] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 853 B/s wr, 180 op/s 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: from='client.24697 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: Saving service iscsi.foo spec with placement count:1 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.741 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 853 B/s wr, 180 op/s 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: from='client.24697 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: Saving service iscsi.foo spec with placement count:1 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.742 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:40.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'sleep 180' 2024-05-03T10:26:41.708 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:41 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:26:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 652 B/s wr, 137 op/s 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: osdmap e79: 8 total, 8 up, 8 in 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: Checking pool "foo" exists for service iscsi.foo 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi043.lxwugw", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-03T10:26:42.868 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi043.lxwugw", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: Deploying daemon iscsi.foo.smithi043.lxwugw on smithi043 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 652 B/s wr, 137 op/s 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: osdmap e79: 8 total, 8 up, 8 in 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.869 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: Checking pool "foo" exists for service iscsi.foo 2024-05-03T10:26:42.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi043.lxwugw", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-03T10:26:42.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi043.lxwugw", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-03T10:26:42.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:42.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: Deploying daemon iscsi.foo.smithi043.lxwugw on smithi043 2024-05-03T10:26:42.870 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 652 B/s wr, 137 op/s 2024-05-03T10:26:42.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: osdmap e79: 8 total, 8 up, 8 in 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: Checking pool "foo" exists for service iscsi.foo 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi043.lxwugw", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi043.lxwugw", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: Deploying daemon iscsi.foo.smithi043.lxwugw on smithi043 2024-05-03T10:26:42.871 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:43.507 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:43] "GET /metrics HTTP/1.1" 200 208009 "" "Prometheus/2.33.4" 2024-05-03T10:26:44.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:44 smithi043 ceph-mon[22404]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 639 B/s wr, 135 op/s 2024-05-03T10:26:44.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:44 smithi043 ceph-mon[22404]: mgrmap e21: y(active, since 2m), standbys: x 2024-05-03T10:26:44.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:44 smithi043 ceph-mon[27243]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 639 B/s wr, 135 op/s 2024-05-03T10:26:44.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:44 smithi043 ceph-mon[27243]: mgrmap e21: y(active, since 2m), standbys: x 2024-05-03T10:26:44.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:44 smithi181 ceph-mon[23393]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 639 B/s wr, 135 op/s 2024-05-03T10:26:44.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:44 smithi181 ceph-mon[23393]: mgrmap e21: y(active, since 2m), standbys: x 2024-05-03T10:26:46.271 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:46 smithi043 ceph-mon[22404]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 520 B/s wr, 109 op/s 2024-05-03T10:26:46.272 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:46 smithi043 ceph-mon[27243]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 520 B/s wr, 109 op/s 2024-05-03T10:26:46.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:46 smithi181 ceph-mon[23393]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 520 B/s wr, 109 op/s 2024-05-03T10:26:47.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:47.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: Adding iSCSI gateway http://:@172.21.15.43:5000 to Dashboard 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi043"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi043"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:47.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/206881126' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/344877038' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2867505315"}]: dispatch 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:47.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: Adding iSCSI gateway http://:@172.21.15.43:5000 to Dashboard 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi043"}]: dispatch 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi043"}]: dispatch 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:47.295 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/206881126' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-03T10:26:47.296 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/344877038' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2867505315"}]: dispatch 2024-05-03T10:26:47.296 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:47 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:47.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: Adding iSCSI gateway http://:@172.21.15.43:5000 to Dashboard 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi043"}]: dispatch 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi043"}]: dispatch 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:47.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:47.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:47.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:47.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-03T10:26:47.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/206881126' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-03T10:26:47.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/344877038' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2867505315"}]: dispatch 2024-05-03T10:26:47.371 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:47 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:48.983 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:48 smithi043 ceph-mon[22404]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 511 B/s wr, 108 op/s 2024-05-03T10:26:48.983 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:48 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/344877038' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2867505315"}]': finished 2024-05-03T10:26:48.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:48 smithi043 ceph-mon[22404]: osdmap e80: 8 total, 8 up, 8 in 2024-05-03T10:26:48.984 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:48 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2984771350' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2999399722"}]: dispatch 2024-05-03T10:26:48.984 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:48 smithi043 ceph-mon[27243]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 511 B/s wr, 108 op/s 2024-05-03T10:26:48.984 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:48 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/344877038' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2867505315"}]': finished 2024-05-03T10:26:48.984 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:48 smithi043 ceph-mon[27243]: osdmap e80: 8 total, 8 up, 8 in 2024-05-03T10:26:48.984 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:48 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2984771350' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2999399722"}]: dispatch 2024-05-03T10:26:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:48 smithi181 ceph-mon[23393]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 511 B/s wr, 108 op/s 2024-05-03T10:26:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:48 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/344877038' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2867505315"}]': finished 2024-05-03T10:26:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:48 smithi181 ceph-mon[23393]: osdmap e80: 8 total, 8 up, 8 in 2024-05-03T10:26:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:48 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2984771350' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2999399722"}]: dispatch 2024-05-03T10:26:49.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:49 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2984771350' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2999399722"}]': finished 2024-05-03T10:26:49.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:49 smithi043 ceph-mon[27243]: osdmap e81: 8 total, 8 up, 8 in 2024-05-03T10:26:49.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:49 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3902982323' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3757009401"}]: dispatch 2024-05-03T10:26:49.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:49 smithi043 ceph-mon[27243]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3757009401"}]: dispatch 2024-05-03T10:26:49.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:49 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2984771350' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2999399722"}]': finished 2024-05-03T10:26:49.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:49 smithi043 ceph-mon[22404]: osdmap e81: 8 total, 8 up, 8 in 2024-05-03T10:26:49.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:49 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3902982323' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3757009401"}]: dispatch 2024-05-03T10:26:49.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:49 smithi043 ceph-mon[22404]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3757009401"}]: dispatch 2024-05-03T10:26:49.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:26:49 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:26:49.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:49 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2984771350' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2999399722"}]': finished 2024-05-03T10:26:49.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:49 smithi181 ceph-mon[23393]: osdmap e81: 8 total, 8 up, 8 in 2024-05-03T10:26:49.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:49 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3902982323' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3757009401"}]: dispatch 2024-05-03T10:26:49.870 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:49 smithi181 ceph-mon[23393]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3757009401"}]: dispatch 2024-05-03T10:26:50.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:49 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:26:49] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:50.927 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[27243]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 140 B/s wr, 1 op/s 2024-05-03T10:26:50.927 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[27243]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3757009401"}]': finished 2024-05-03T10:26:50.927 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[27243]: osdmap e82: 8 total, 8 up, 8 in 2024-05-03T10:26:50.927 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:50.927 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1220849889' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3299376245"}]: dispatch 2024-05-03T10:26:50.927 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[22404]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 140 B/s wr, 1 op/s 2024-05-03T10:26:50.927 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[22404]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3757009401"}]': finished 2024-05-03T10:26:50.928 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[22404]: osdmap e82: 8 total, 8 up, 8 in 2024-05-03T10:26:50.928 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:50.928 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:50 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1220849889' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3299376245"}]: dispatch 2024-05-03T10:26:51.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:50 smithi181 ceph-mon[23393]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 140 B/s wr, 1 op/s 2024-05-03T10:26:51.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:50 smithi181 ceph-mon[23393]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3757009401"}]': finished 2024-05-03T10:26:51.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:50 smithi181 ceph-mon[23393]: osdmap e82: 8 total, 8 up, 8 in 2024-05-03T10:26:51.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:51.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:50 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1220849889' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3299376245"}]: dispatch 2024-05-03T10:26:52.244 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:51 smithi043 ceph-mon[27243]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 1 op/s 2024-05-03T10:26:52.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:51 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1220849889' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3299376245"}]': finished 2024-05-03T10:26:52.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:51 smithi043 ceph-mon[27243]: osdmap e83: 8 total, 8 up, 8 in 2024-05-03T10:26:52.245 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:51 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1762245852' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2999399722"}]: dispatch 2024-05-03T10:26:52.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:51 smithi043 ceph-mon[22404]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 1 op/s 2024-05-03T10:26:52.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:51 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1220849889' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3299376245"}]': finished 2024-05-03T10:26:52.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:51 smithi043 ceph-mon[22404]: osdmap e83: 8 total, 8 up, 8 in 2024-05-03T10:26:52.245 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:51 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1762245852' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2999399722"}]: dispatch 2024-05-03T10:26:52.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:51 smithi181 ceph-mon[23393]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 1 op/s 2024-05-03T10:26:52.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:51 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1220849889' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3299376245"}]': finished 2024-05-03T10:26:52.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:51 smithi181 ceph-mon[23393]: osdmap e83: 8 total, 8 up, 8 in 2024-05-03T10:26:52.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:51 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1762245852' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2999399722"}]: dispatch 2024-05-03T10:26:52.981 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:52 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:26:52] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:26:53.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:26:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:53] "GET /metrics HTTP/1.1" 200 208009 "" "Prometheus/2.33.4" 2024-05-03T10:26:53.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1762245852' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2999399722"}]': finished 2024-05-03T10:26:53.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[22404]: osdmap e84: 8 total, 8 up, 8 in 2024-05-03T10:26:53.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1317264459' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2216262703"}]: dispatch 2024-05-03T10:26:53.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1762245852' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2999399722"}]': finished 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[27243]: osdmap e84: 8 total, 8 up, 8 in 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1317264459' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2216262703"}]: dispatch 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:53.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:53.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:53.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:53.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:52 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1762245852' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2999399722"}]': finished 2024-05-03T10:26:53.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:52 smithi181 ceph-mon[23393]: osdmap e84: 8 total, 8 up, 8 in 2024-05-03T10:26:53.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:52 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1317264459' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2216262703"}]: dispatch 2024-05-03T10:26:53.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:53.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:53.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:26:53.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:26:53.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:26:53.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:26:53.792 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:26:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=warn ts=2024-05-03T10:26:53.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.181:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.181 because it doesn't contain any IP SANs" 2024-05-03T10:26:53.792 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:26:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=warn ts=2024-05-03T10:26:53.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.43:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.43 because it doesn't contain any IP SANs" 2024-05-03T10:26:54.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:53 smithi043 ceph-mon[27243]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:26:54.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:53 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1317264459' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2216262703"}]': finished 2024-05-03T10:26:54.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:53 smithi043 ceph-mon[27243]: osdmap e85: 8 total, 8 up, 8 in 2024-05-03T10:26:54.294 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:53 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/684903489' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/418629400"}]: dispatch 2024-05-03T10:26:54.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:53 smithi043 ceph-mon[22404]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:26:54.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:53 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1317264459' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2216262703"}]': finished 2024-05-03T10:26:54.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:53 smithi043 ceph-mon[22404]: osdmap e85: 8 total, 8 up, 8 in 2024-05-03T10:26:54.294 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:53 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/684903489' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/418629400"}]: dispatch 2024-05-03T10:26:54.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:53 smithi181 ceph-mon[23393]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:26:54.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:53 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1317264459' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/2216262703"}]': finished 2024-05-03T10:26:54.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:53 smithi181 ceph-mon[23393]: osdmap e85: 8 total, 8 up, 8 in 2024-05-03T10:26:54.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:53 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/684903489' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/418629400"}]: dispatch 2024-05-03T10:26:55.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:54 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/684903489' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/418629400"}]': finished 2024-05-03T10:26:55.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:54 smithi043 ceph-mon[22404]: osdmap e86: 8 total, 8 up, 8 in 2024-05-03T10:26:55.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:54 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1823265596' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2358376213"}]: dispatch 2024-05-03T10:26:55.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:54 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/684903489' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/418629400"}]': finished 2024-05-03T10:26:55.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:54 smithi043 ceph-mon[27243]: osdmap e86: 8 total, 8 up, 8 in 2024-05-03T10:26:55.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:54 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1823265596' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2358376213"}]: dispatch 2024-05-03T10:26:55.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:54 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/684903489' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/418629400"}]': finished 2024-05-03T10:26:55.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:54 smithi181 ceph-mon[23393]: osdmap e86: 8 total, 8 up, 8 in 2024-05-03T10:26:55.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:54 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1823265596' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2358376213"}]: dispatch 2024-05-03T10:26:56.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:55 smithi043 ceph-mon[27243]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:26:56.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:55 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1823265596' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2358376213"}]': finished 2024-05-03T10:26:56.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:55 smithi043 ceph-mon[27243]: osdmap e87: 8 total, 8 up, 8 in 2024-05-03T10:26:56.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:55 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2972725675' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1017574898"}]: dispatch 2024-05-03T10:26:56.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:55 smithi043 ceph-mon[22404]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:26:56.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:55 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1823265596' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2358376213"}]': finished 2024-05-03T10:26:56.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:55 smithi043 ceph-mon[22404]: osdmap e87: 8 total, 8 up, 8 in 2024-05-03T10:26:56.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:55 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2972725675' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1017574898"}]: dispatch 2024-05-03T10:26:56.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:55 smithi181 ceph-mon[23393]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:26:56.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:55 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1823265596' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/2358376213"}]': finished 2024-05-03T10:26:56.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:55 smithi181 ceph-mon[23393]: osdmap e87: 8 total, 8 up, 8 in 2024-05-03T10:26:56.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:55 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2972725675' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1017574898"}]: dispatch 2024-05-03T10:26:57.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:57 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2972725675' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1017574898"}]': finished 2024-05-03T10:26:57.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:57 smithi181 ceph-mon[23393]: osdmap e88: 8 total, 8 up, 8 in 2024-05-03T10:26:57.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:57 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:26:57.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:57 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/308961920' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2358376213"}]: dispatch 2024-05-03T10:26:57.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:57 smithi181 ceph-mon[23393]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2358376213"}]: dispatch 2024-05-03T10:26:57.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:56 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2972725675' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1017574898"}]': finished 2024-05-03T10:26:57.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:57 smithi043 ceph-mon[27243]: osdmap e88: 8 total, 8 up, 8 in 2024-05-03T10:26:57.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:57 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:26:57.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:57 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/308961920' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2358376213"}]: dispatch 2024-05-03T10:26:57.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:57 smithi043 ceph-mon[27243]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2358376213"}]: dispatch 2024-05-03T10:26:57.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:56 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2972725675' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1017574898"}]': finished 2024-05-03T10:26:57.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:56 smithi043 ceph-mon[22404]: osdmap e88: 8 total, 8 up, 8 in 2024-05-03T10:26:57.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:56 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:26:57.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:56 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/308961920' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2358376213"}]: dispatch 2024-05-03T10:26:57.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:56 smithi043 ceph-mon[22404]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2358376213"}]: dispatch 2024-05-03T10:26:58.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:58 smithi043 ceph-mon[27243]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:58.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:58 smithi043 ceph-mon[27243]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2358376213"}]': finished 2024-05-03T10:26:58.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:58 smithi043 ceph-mon[27243]: osdmap e89: 8 total, 8 up, 8 in 2024-05-03T10:26:58.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:58 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1596295038' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3170124346"}]: dispatch 2024-05-03T10:26:58.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:58 smithi043 ceph-mon[22404]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:58.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:58 smithi043 ceph-mon[22404]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2358376213"}]': finished 2024-05-03T10:26:58.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:58 smithi043 ceph-mon[22404]: osdmap e89: 8 total, 8 up, 8 in 2024-05-03T10:26:58.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:58 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1596295038' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3170124346"}]: dispatch 2024-05-03T10:26:58.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:58 smithi181 ceph-mon[23393]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:26:58.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:58 smithi181 ceph-mon[23393]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/2358376213"}]': finished 2024-05-03T10:26:58.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:58 smithi181 ceph-mon[23393]: osdmap e89: 8 total, 8 up, 8 in 2024-05-03T10:26:58.370 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:58 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1596295038' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3170124346"}]: dispatch 2024-05-03T10:26:59.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:59 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1596295038' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3170124346"}]': finished 2024-05-03T10:26:59.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:59 smithi043 ceph-mon[22404]: osdmap e90: 8 total, 8 up, 8 in 2024-05-03T10:26:59.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:26:59 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/516247971' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1256461377"}]: dispatch 2024-05-03T10:26:59.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:59 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1596295038' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3170124346"}]': finished 2024-05-03T10:26:59.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:59 smithi043 ceph-mon[27243]: osdmap e90: 8 total, 8 up, 8 in 2024-05-03T10:26:59.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:26:59 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/516247971' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1256461377"}]: dispatch 2024-05-03T10:26:59.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:59 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1596295038' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/3170124346"}]': finished 2024-05-03T10:26:59.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:59 smithi181 ceph-mon[23393]: osdmap e90: 8 total, 8 up, 8 in 2024-05-03T10:26:59.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:26:59 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/516247971' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1256461377"}]: dispatch 2024-05-03T10:26:59.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:26:59 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:26:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:27:00.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:00 smithi043 ceph-mon[22404]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:00.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:00 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/516247971' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1256461377"}]': finished 2024-05-03T10:27:00.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:00 smithi043 ceph-mon[22404]: osdmap e91: 8 total, 8 up, 8 in 2024-05-03T10:27:00.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:00 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/522568101' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/992331217"}]: dispatch 2024-05-03T10:27:00.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:00 smithi043 ceph-mon[27243]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:00.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:00 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/516247971' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1256461377"}]': finished 2024-05-03T10:27:00.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:00 smithi043 ceph-mon[27243]: osdmap e91: 8 total, 8 up, 8 in 2024-05-03T10:27:00.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:00 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/522568101' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/992331217"}]: dispatch 2024-05-03T10:27:00.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:00 smithi181 ceph-mon[23393]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:00.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:00 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/516247971' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1256461377"}]': finished 2024-05-03T10:27:00.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:00 smithi181 ceph-mon[23393]: osdmap e91: 8 total, 8 up, 8 in 2024-05-03T10:27:00.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:00 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/522568101' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/992331217"}]: dispatch 2024-05-03T10:27:00.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:00 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:27:00] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:27:01.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/522568101' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/992331217"}]': finished 2024-05-03T10:27:01.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[22404]: osdmap e92: 8 total, 8 up, 8 in 2024-05-03T10:27:01.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2792989335' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/992331217"}]: dispatch 2024-05-03T10:27:01.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:01.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:01.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/522568101' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/992331217"}]': finished 2024-05-03T10:27:01.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[27243]: osdmap e92: 8 total, 8 up, 8 in 2024-05-03T10:27:01.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2792989335' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/992331217"}]: dispatch 2024-05-03T10:27:01.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:01.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:01 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:01.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:01 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/522568101' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6801/992331217"}]': finished 2024-05-03T10:27:01.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:01 smithi181 ceph-mon[23393]: osdmap e92: 8 total, 8 up, 8 in 2024-05-03T10:27:01.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:01 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2792989335' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/992331217"}]: dispatch 2024-05-03T10:27:01.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:01 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:01.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:01 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:02.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[22404]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:02.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2792989335' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/992331217"}]': finished 2024-05-03T10:27:02.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[22404]: osdmap e93: 8 total, 8 up, 8 in 2024-05-03T10:27:02.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1921839476' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1526951672"}]: dispatch 2024-05-03T10:27:02.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[22404]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1526951672"}]: dispatch 2024-05-03T10:27:02.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[27243]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:02.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2792989335' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/992331217"}]': finished 2024-05-03T10:27:02.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[27243]: osdmap e93: 8 total, 8 up, 8 in 2024-05-03T10:27:02.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1921839476' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1526951672"}]: dispatch 2024-05-03T10:27:02.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:02 smithi043 ceph-mon[27243]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1526951672"}]: dispatch 2024-05-03T10:27:02.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:02 smithi181 ceph-mon[23393]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:02.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:02 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2792989335' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:6800/992331217"}]': finished 2024-05-03T10:27:02.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:02 smithi181 ceph-mon[23393]: osdmap e93: 8 total, 8 up, 8 in 2024-05-03T10:27:02.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:02 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1921839476' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1526951672"}]: dispatch 2024-05-03T10:27:02.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:02 smithi181 ceph-mon[23393]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1526951672"}]: dispatch 2024-05-03T10:27:03.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:03 smithi181 ceph-mon[23393]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1526951672"}]': finished 2024-05-03T10:27:03.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:03 smithi181 ceph-mon[23393]: osdmap e94: 8 total, 8 up, 8 in 2024-05-03T10:27:03.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:03 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2575839980' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/868367844"}]: dispatch 2024-05-03T10:27:03.507 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:03 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:03] "GET /metrics HTTP/1.1" 200 208031 "" "Prometheus/2.33.4" 2024-05-03T10:27:03.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:03 smithi043 ceph-mon[27243]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1526951672"}]': finished 2024-05-03T10:27:03.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:03 smithi043 ceph-mon[27243]: osdmap e94: 8 total, 8 up, 8 in 2024-05-03T10:27:03.507 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:03 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2575839980' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/868367844"}]: dispatch 2024-05-03T10:27:03.508 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:03 smithi043 ceph-mon[22404]: from='client.? ' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/1526951672"}]': finished 2024-05-03T10:27:03.508 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:03 smithi043 ceph-mon[22404]: osdmap e94: 8 total, 8 up, 8 in 2024-05-03T10:27:03.508 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:03 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2575839980' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/868367844"}]: dispatch 2024-05-03T10:27:03.792 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:27:03 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=error ts=2024-05-03T10:27:03.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.43:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.43 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.181:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.181 because it doesn't contain any IP SANs" 2024-05-03T10:27:04.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:04 smithi181 ceph-mon[23393]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:04.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:04 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2575839980' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/868367844"}]': finished 2024-05-03T10:27:04.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:04 smithi181 ceph-mon[23393]: osdmap e95: 8 total, 8 up, 8 in 2024-05-03T10:27:04.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:04 smithi043 ceph-mon[27243]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:04.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:04 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2575839980' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/868367844"}]': finished 2024-05-03T10:27:04.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:04 smithi043 ceph-mon[27243]: osdmap e95: 8 total, 8 up, 8 in 2024-05-03T10:27:04.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:04 smithi043 ceph-mon[22404]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:04.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:04 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2575839980' entity='client.iscsi.foo.smithi043.lxwugw' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.43:0/868367844"}]': finished 2024-05-03T10:27:04.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:04 smithi043 ceph-mon[22404]: osdmap e95: 8 total, 8 up, 8 in 2024-05-03T10:27:06.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:06 smithi181 ceph-mon[23393]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-03T10:27:06.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:06 smithi043 ceph-mon[22404]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-03T10:27:06.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:06 smithi043 ceph-mon[27243]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-03T10:27:07.924 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:07 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:07.924 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:07 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:08.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:07 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:08.846 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:08 smithi043 ceph-mon[22404]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:08.846 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:08 smithi043 ceph-mon[27243]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:08 smithi181 ceph-mon[23393]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:09.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:08 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:27:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:27:09.840 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:27:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:27:10.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:09 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:10.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:09 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:10.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:09 smithi043 ceph-mon[27243]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-03T10:27:10.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:09 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:10.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:09 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:10.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:09 smithi043 ceph-mon[22404]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-03T10:27:10.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:09 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:10.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:09 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:10.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:09 smithi181 ceph-mon[23393]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-03T10:27:13.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:12 smithi043 ceph-mon[27243]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-03T10:27:13.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:12 smithi043 ceph-mon[22404]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-03T10:27:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:12 smithi181 ceph-mon[23393]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-03T10:27:13.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:13] "GET /metrics HTTP/1.1" 200 208046 "" "Prometheus/2.33.4" 2024-05-03T10:27:15.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:14 smithi043 ceph-mon[22404]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-03T10:27:15.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:14 smithi043 ceph-mon[27243]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-03T10:27:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:14 smithi181 ceph-mon[23393]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-03T10:27:16.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:16 smithi181 ceph-mon[23393]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-05-03T10:27:17.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:16 smithi043 ceph-mon[22404]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-05-03T10:27:17.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:16 smithi043 ceph-mon[27243]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-05-03T10:27:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:17 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:17 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:18.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:17 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:19.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:18 smithi043 ceph-mon[22404]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:19.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:18 smithi043 ceph-mon[27243]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:18 smithi181 ceph-mon[23393]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:19.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:27:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:27:20.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:20 smithi181 ceph-mon[23393]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:20.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:20 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:20.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:20 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:20 smithi043 ceph-mon[22404]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:20 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:20 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:21.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:20 smithi043 ceph-mon[27243]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:21.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:20 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:21.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:20 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:21.043 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:20 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:27:20] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:27:23.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:22 smithi043 ceph-mon[22404]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:23.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:22 smithi043 ceph-mon[27243]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:23.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:22 smithi181 ceph-mon[23393]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:23.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:23] "GET /metrics HTTP/1.1" 200 208046 "" "Prometheus/2.33.4" 2024-05-03T10:27:25.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:24 smithi043 ceph-mon[22404]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:25.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:24 smithi043 ceph-mon[27243]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:25.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:24 smithi181 ceph-mon[23393]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:27.016 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:26 smithi043 ceph-mon[22404]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:27.016 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:26 smithi043 ceph-mon[27243]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:27.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:26 smithi181 ceph-mon[23393]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:27.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:27 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:27.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:27:27.965 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:27:27.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:27 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:27.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:27:27.966 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:27:27.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:27 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:27.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:27:27.967 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:27:28.947 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:28 smithi043 ceph-mon[22404]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:28.947 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:28 smithi043 ceph-mon[27243]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:28 smithi181 ceph-mon[23393]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:29.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:28 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:27:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:27:29.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:27:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:27:30.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:29 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:30.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:29 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:30.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:29 smithi043 ceph-mon[27243]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:30.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:29 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:30.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:29 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:30.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:29 smithi043 ceph-mon[22404]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:30.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:29 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:30.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:29 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:30.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:29 smithi181 ceph-mon[23393]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:33.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:32 smithi043 ceph-mon[22404]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:33.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:32 smithi043 ceph-mon[27243]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:33.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:32 smithi181 ceph-mon[23393]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:33.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:33 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:33] "GET /metrics HTTP/1.1" 200 208009 "" "Prometheus/2.33.4" 2024-05-03T10:27:35.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:34 smithi043 ceph-mon[22404]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:35.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:34 smithi043 ceph-mon[27243]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:35.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:34 smithi181 ceph-mon[23393]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:37.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:36 smithi043 ceph-mon[22404]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:37.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:36 smithi043 ceph-mon[27243]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:37.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:36 smithi181 ceph-mon[23393]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:37.767 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:37 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:38.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:37 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:38.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:37 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:39.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:38 smithi043 ceph-mon[22404]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:39.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:38 smithi043 ceph-mon[27243]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:39.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:38 smithi181 ceph-mon[23393]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:39.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:27:39 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:27:41.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:40 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:27:40] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:27:41.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:40 smithi043 ceph-mon[27243]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:41.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:41.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:40 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:41.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:40 smithi043 ceph-mon[22404]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:41.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:41.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:40 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:41.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:40 smithi181 ceph-mon[23393]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:41.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:41.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:40 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:43.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:42 smithi043 ceph-mon[22404]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-03T10:27:43.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:42 smithi043 ceph-mon[27243]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-03T10:27:43.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:42 smithi181 ceph-mon[23393]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-03T10:27:43.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:43] "GET /metrics HTTP/1.1" 200 207960 "" "Prometheus/2.33.4" 2024-05-03T10:27:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:44 smithi043 ceph-mon[22404]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:45.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:44 smithi043 ceph-mon[27243]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:45.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:44 smithi181 ceph-mon[23393]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:46.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:46 smithi043 ceph-mon[22404]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:46.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:46 smithi043 ceph-mon[27243]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:47.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:46 smithi181 ceph-mon[23393]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:48.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:47 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:48.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:47 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:47 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:48.921 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:48 smithi043 ceph-mon[27243]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:48.921 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:48 smithi043 ceph-mon[22404]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:48 smithi181 ceph-mon[23393]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:49.180 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:49 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:27:49] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:27:49.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:27:49 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:27:50.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:50 smithi043 ceph-mon[22404]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:50.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:50.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:50.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:50 smithi043 ceph-mon[27243]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:50.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:50.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:50.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:50 smithi181 ceph-mon[23393]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:50.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:50.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:53.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[22404]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:53.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:27:53.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:27:53.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:27:53.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:53.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[27243]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:53.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:27:53.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:27:53.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:27:53.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:52 smithi181 ceph-mon[23393]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:27:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:27:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:27:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:27:53.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:27:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:53] "GET /metrics HTTP/1.1" 200 207960 "" "Prometheus/2.33.4" 2024-05-03T10:27:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:54 smithi043 ceph-mon[22404]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:55.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:54 smithi043 ceph-mon[27243]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:54 smithi181 ceph-mon[23393]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:27:56.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:56 smithi181 ceph-mon[23393]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:57.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:56 smithi043 ceph-mon[22404]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:57.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:56 smithi043 ceph-mon[27243]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:58.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:57 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:58.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:57 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:58.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:57 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:27:58.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:27:58 smithi181 ceph-mon[23393]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:59.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:27:58 smithi043 ceph-mon[22404]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:27:58 smithi043 ceph-mon[27243]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:27:59.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:27:59 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:27:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:28:01.034 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:00 smithi043 ceph-mon[27243]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:01.035 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:00 smithi043 ceph-mon[22404]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:01.035 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:00 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:28:00] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:28:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:00 smithi181 ceph-mon[23393]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:02.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:01 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:02.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:01 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:02.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:01 smithi043 ceph-mon[22404]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:02.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:01 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:02.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:01 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:02.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:01 smithi043 ceph-mon[27243]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:02.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:01 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:02.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:01 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:02.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:01 smithi181 ceph-mon[23393]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:03.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:03 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:03] "GET /metrics HTTP/1.1" 200 207966 "" "Prometheus/2.33.4" 2024-05-03T10:28:04.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:04 smithi043 ceph-mon[22404]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:04.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:04 smithi043 ceph-mon[27243]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:04.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:04 smithi181 ceph-mon[23393]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:06.282 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:06 smithi043 ceph-mon[22404]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:06.283 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:06 smithi043 ceph-mon[27243]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:06.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:06 smithi181 ceph-mon[23393]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:08.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:07 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:08.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:07 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:08.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:07 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:09.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:08 smithi043 ceph-mon[22404]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:09.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:08 smithi043 ceph-mon[27243]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:08 smithi181 ceph-mon[23393]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:09.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:09 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:28:09] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:28:09.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:28:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:28:10.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:10 smithi043 ceph-mon[27243]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:10.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:10.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:10.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:10 smithi043 ceph-mon[22404]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:10.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:10.543 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:10.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:10 smithi181 ceph-mon[23393]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:10.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:10.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:13.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:12 smithi043 ceph-mon[22404]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:13.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:12 smithi043 ceph-mon[27243]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:12 smithi181 ceph-mon[23393]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:13.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:13] "GET /metrics HTTP/1.1" 200 207967 "" "Prometheus/2.33.4" 2024-05-03T10:28:15.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:14 smithi043 ceph-mon[22404]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:15.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:14 smithi043 ceph-mon[27243]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:14 smithi181 ceph-mon[23393]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:16.769 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:16 smithi181 ceph-mon[23393]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:17.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:16 smithi043 ceph-mon[22404]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:17.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:16 smithi043 ceph-mon[27243]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:17 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:17 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:18.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:17 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:19.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:18 smithi043 ceph-mon[22404]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:19.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:18 smithi043 ceph-mon[27243]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:19.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:18 smithi181 ceph-mon[23393]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:19.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:28:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:28:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:20 smithi043 ceph-mon[22404]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:21.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:20 smithi043 ceph-mon[27243]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:21.043 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:20 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:28:20] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:28:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:20 smithi181 ceph-mon[23393]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:22.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:21 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:22.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:21 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:22.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:21 smithi181 ceph-mon[23393]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:22.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:21 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:22.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:21 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:22.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:21 smithi043 ceph-mon[22404]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:22.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:21 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:22.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:21 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:22.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:21 smithi043 ceph-mon[27243]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:23.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:23] "GET /metrics HTTP/1.1" 200 207967 "" "Prometheus/2.33.4" 2024-05-03T10:28:24.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:24 smithi043 ceph-mon[22404]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:24.327 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:24 smithi043 ceph-mon[27243]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:24.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:24 smithi181 ceph-mon[23393]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:26.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:26 smithi043 ceph-mon[22404]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:26.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:26 smithi043 ceph-mon[27243]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:26.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:26 smithi181 ceph-mon[23393]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:27 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:28:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:28:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:27 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:28:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:28:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:27 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:28:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:28:29.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:28 smithi043 ceph-mon[27243]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:28 smithi043 ceph-mon[22404]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:28 smithi181 ceph-mon[23393]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:29.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:29 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:28:29] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:28:29.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:28:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:28:30.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:30 smithi043 ceph-mon[22404]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:30.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:30.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:30.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:30 smithi043 ceph-mon[27243]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:30.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:30.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:30.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:30 smithi181 ceph-mon[23393]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:30.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:30.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:32.750 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:32 smithi043 ceph-mon[22404]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:32.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:32 smithi043 ceph-mon[27243]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:33.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:32 smithi181 ceph-mon[23393]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:33.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:33 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:33] "GET /metrics HTTP/1.1" 200 207990 "" "Prometheus/2.33.4" 2024-05-03T10:28:35.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:34 smithi043 ceph-mon[22404]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:35.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:34 smithi043 ceph-mon[27243]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:35.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:34 smithi181 ceph-mon[23393]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:36.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:36 smithi181 ceph-mon[23393]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:37.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:36 smithi043 ceph-mon[27243]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:37.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:36 smithi043 ceph-mon[22404]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:37.866 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:37 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:38.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:37 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:38.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:37 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:38.937 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:38 smithi181 ceph-mon[23393]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:39.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:38 smithi043 ceph-mon[27243]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:39.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:38 smithi043 ceph-mon[22404]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:39.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:28:39 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:28:40.979 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:40 smithi043 ceph-mon[27243]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:40.980 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:40 smithi043 ceph-mon[22404]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:40.980 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:40 smithi181 ceph-mon[23393]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:41.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:40 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:28:40] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:28:42.227 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:41 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:42.227 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:41 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:42.227 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:41 smithi043 ceph-mon[22404]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:42.227 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:41 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:42.227 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:41 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:42.227 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:41 smithi043 ceph-mon[27243]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:42.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:41 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:42.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:41 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:42.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:41 smithi181 ceph-mon[23393]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:43.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:43] "GET /metrics HTTP/1.1" 200 207969 "" "Prometheus/2.33.4" 2024-05-03T10:28:44.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:44 smithi043 ceph-mon[22404]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:44.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:44 smithi043 ceph-mon[27243]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:44.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:44 smithi181 ceph-mon[23393]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:46.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:46 smithi043 ceph-mon[27243]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:46.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:46 smithi043 ceph-mon[22404]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:46.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:46 smithi181 ceph-mon[23393]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:48.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:47 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:48.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:47 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:47 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:49.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:48 smithi043 ceph-mon[22404]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:49.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:48 smithi043 ceph-mon[27243]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:48 smithi181 ceph-mon[23393]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:49.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:28:49 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:28:49.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:49 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:28:49] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:28:50.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:50 smithi181 ceph-mon[23393]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:50.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:50.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:50.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:50 smithi043 ceph-mon[27243]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:50.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:50.709 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:50.709 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:50 smithi043 ceph-mon[22404]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:50.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:50.710 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:53.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[22404]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:53.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:28:53.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:28:53.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:28:53.015 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:53.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[27243]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:53.015 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:28:53.016 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:28:53.016 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:28:53.016 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:52 smithi181 ceph-mon[23393]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:28:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:28:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:28:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:28:53.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:28:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:53] "GET /metrics HTTP/1.1" 200 207969 "" "Prometheus/2.33.4" 2024-05-03T10:28:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:54 smithi043 ceph-mon[22404]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:55.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:54 smithi043 ceph-mon[27243]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:54 smithi181 ceph-mon[23393]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:28:56.798 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:56 smithi181 ceph-mon[23393]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:57.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:56 smithi043 ceph-mon[22404]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:57.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:56 smithi043 ceph-mon[27243]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:58.027 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:57 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:58.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:57 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:58.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:57 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:28:58.854 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:28:58 smithi181 ceph-mon[23393]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:59.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:28:58 smithi043 ceph-mon[22404]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:28:58 smithi043 ceph-mon[27243]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:28:59.608 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:28:59 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:28:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:29:01.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:00 smithi043 ceph-mon[22404]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:01.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:00 smithi043 ceph-mon[27243]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:01.086 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:00 smithi181 ceph-mon[23393]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:01.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:01 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:29:01] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:29:02.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:02 smithi181 ceph-mon[23393]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:02.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:02.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:02.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:02 smithi043 ceph-mon[22404]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:02.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:02.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:02.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:02 smithi043 ceph-mon[27243]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:02.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:02.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:03.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:03 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:03] "GET /metrics HTTP/1.1" 200 207974 "" "Prometheus/2.33.4" 2024-05-03T10:29:04.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:04 smithi181 ceph-mon[23393]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:04.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:04 smithi043 ceph-mon[27243]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:04.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:04 smithi043 ceph-mon[22404]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:06.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:06 smithi181 ceph-mon[23393]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:06.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:06 smithi043 ceph-mon[22404]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:06.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:06 smithi043 ceph-mon[27243]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:08.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:07 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:08.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:07 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:08.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:07 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:09.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:08 smithi043 ceph-mon[22404]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:09.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:08 smithi043 ceph-mon[27243]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:08 smithi181 ceph-mon[23393]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:09.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:09 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:29:09] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:29:09.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:29:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:29:10.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:10 smithi043 ceph-mon[27243]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:10.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:10.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:10.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:10 smithi043 ceph-mon[22404]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:10.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:10.793 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:10.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:10 smithi181 ceph-mon[23393]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:10.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:10.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:12.782 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:12 smithi043 ceph-mon[22404]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:12.783 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:12 smithi043 ceph-mon[27243]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:12 smithi181 ceph-mon[23393]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:13.462 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:13] "GET /metrics HTTP/1.1" 200 207927 "" "Prometheus/2.33.4" 2024-05-03T10:29:15.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:14 smithi043 ceph-mon[22404]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:15.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:14 smithi043 ceph-mon[27243]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:14 smithi181 ceph-mon[23393]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:16.936 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:16 smithi181 ceph-mon[23393]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:17.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:16 smithi043 ceph-mon[22404]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:17.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:16 smithi043 ceph-mon[27243]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:18.022 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:17 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:17 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:18.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:17 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:18.942 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:18 smithi181 ceph-mon[23393]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:19.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:18 smithi043 ceph-mon[22404]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:19.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:18 smithi043 ceph-mon[27243]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:19.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:29:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:29:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:20 smithi043 ceph-mon[22404]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:21.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:20 smithi043 ceph-mon[27243]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:20 smithi181 ceph-mon[23393]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:21.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:21 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:29:21] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:29:22.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:22 smithi043 ceph-mon[22404]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:22.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:22.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:22.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:22 smithi043 ceph-mon[27243]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:22.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:22.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:22.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:22 smithi181 ceph-mon[23393]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:22.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:22.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:23.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:23] "GET /metrics HTTP/1.1" 200 207927 "" "Prometheus/2.33.4" 2024-05-03T10:29:24.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:24 smithi043 ceph-mon[22404]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:24.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:24 smithi043 ceph-mon[27243]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:24.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:24 smithi181 ceph-mon[23393]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:26.542 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:26 smithi043 ceph-mon[27243]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:26.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:26 smithi043 ceph-mon[22404]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:26.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:26 smithi181 ceph-mon[23393]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:27 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:29:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:29:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:27 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:29:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:29:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:27 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:29:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:29:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:28 smithi043 ceph-mon[22404]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:29.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:28 smithi043 ceph-mon[27243]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:28 smithi181 ceph-mon[23393]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:29.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:29:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:29:30.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:29 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:29:29] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:29:30.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:30 smithi181 ceph-mon[23393]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:30.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:30.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:31.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:30 smithi043 ceph-mon[27243]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:31.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:31.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:30 smithi043 ceph-mon[22404]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:31.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:31.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:32.749 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:32 smithi043 ceph-mon[27243]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:32.749 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:32 smithi043 ceph-mon[22404]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:33.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:32 smithi181 ceph-mon[23393]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:33.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:33 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:33] "GET /metrics HTTP/1.1" 200 207920 "" "Prometheus/2.33.4" 2024-05-03T10:29:35.010 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:34 smithi043 ceph-mon[22404]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:35.010 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:34 smithi043 ceph-mon[27243]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:35.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:34 smithi181 ceph-mon[23393]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:36.958 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:36 smithi181 ceph-mon[23393]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:37.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:36 smithi043 ceph-mon[22404]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:37.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:36 smithi043 ceph-mon[27243]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:37.948 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:37 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:38.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:37 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:38.045 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:37 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:39.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:38 smithi043 ceph-mon[27243]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:39.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:38 smithi043 ceph-mon[22404]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:39.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:38 smithi181 ceph-mon[23393]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:39.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:29:39 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:29:41.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:40 smithi043 ceph-mon[22404]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:41.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:40 smithi043 ceph-mon[27243]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:41.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:40 smithi181 ceph-mon[23393]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:41.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:41 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:29:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:29:42.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:42 smithi043 ceph-mon[22404]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:42.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:42.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:42.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:42 smithi043 ceph-mon[27243]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:42.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:42.543 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:42.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:42 smithi181 ceph-mon[23393]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:42.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:42.620 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:43.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:43] "GET /metrics HTTP/1.1" 200 207923 "" "Prometheus/2.33.4" 2024-05-03T10:29:44.439 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:44 smithi043 ceph-mon[22404]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:44.440 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:44 smithi043 ceph-mon[27243]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:44.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-05-03T10:29:44.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:44 smithi181 ceph-mon[23393]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:46.369 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:46 smithi043 ceph-mon[22404]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:46.370 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:46 smithi043 ceph-mon[27243]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:46.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:46 smithi181 ceph-mon[23393]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:47.198 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-05-03T10:29:47.716 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:47 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:47.716 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:47 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:47.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:47 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:49.024 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:48 smithi043 ceph-mon[27243]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:49.025 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:48 smithi043 ceph-mon[22404]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:48 smithi181 ceph-mon[23393]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:49.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:29:49 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:29:49.912 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:49 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:29:49] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:29:49.936 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph config set global log_to_journald false --force' 2024-05-03T10:29:50.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:50 smithi043 ceph-mon[22404]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:50.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:50.953 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:50.954 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:50 smithi043 ceph-mon[27243]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:50.954 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:50.954 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:50 smithi181 ceph-mon[23393]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:51.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:51.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:52.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph orch ps' 2024-05-03T10:29:52.923 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:52 smithi043 ceph-mon[27243]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:52.924 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:29:52.924 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:29:52.924 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:52 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:29:52.924 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:52 smithi043 ceph-mon[22404]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:52.924 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:29:52.925 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:29:52.925 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:52 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:29:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:52 smithi181 ceph-mon[23393]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:29:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T10:29:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:52 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T10:29:53.239 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:29:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:53] "GET /metrics HTTP/1.1" 200 207923 "" "Prometheus/2.33.4" 2024-05-03T10:29:54.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:53 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:54.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:53 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:54.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:53 smithi043 ceph-mon[27243]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:54.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:53 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:54.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:53 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:54.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:53 smithi043 ceph-mon[22404]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:54.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:53 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:54.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:53 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:29:54.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:53 smithi181 ceph-mon[23393]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:29:55.487 INFO:teuthology.orchestra.run.smithi043.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-03T10:29:55.487 INFO:teuthology.orchestra.run.smithi043.stdout:agent.smithi043 smithi043 running 5s ago 9m - - 2024-05-03T10:29:55.487 INFO:teuthology.orchestra.run.smithi043.stdout:agent.smithi181 smithi181 running 14s ago 9m - - 2024-05-03T10:29:55.487 INFO:teuthology.orchestra.run.smithi043.stdout:alertmanager.a smithi043 *:9093,9094 running (4m) 5s ago 4m 28.3M - ba2b418f427c 43fbcd25b859 2024-05-03T10:29:55.487 INFO:teuthology.orchestra.run.smithi043.stdout:grafana.a smithi181 *:3000 running (4m) 14s ago 4m 55.0M - 8.3.5 dad864ee21e9 15a635565ef8 2024-05-03T10:29:55.487 INFO:teuthology.orchestra.run.smithi043.stdout:iscsi.foo.smithi043.lxwugw smithi043 running (3m) 5s ago 3m 46.7M - 3.5 e1d6a67b021e 89b7cc11efa9 2024-05-03T10:29:55.487 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.x smithi181 *:8443 running (9m) 14s ago 9m 412M - 17.2.0 e1d6a67b021e f42a19b78d32 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.y smithi043 *:9283 running (10m) 5s ago 10m 464M - 17.2.0 e1d6a67b021e e925f3a624c5 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a smithi043 running (10m) 5s ago 10m 50.0M 2048M 17.2.0 e1d6a67b021e 92837a4d1736 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b smithi181 running (9m) 14s ago 9m 37.4M 2048M 17.2.0 e1d6a67b021e 5e27fbb56907 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c smithi043 running (9m) 5s ago 9m 37.0M 2048M 17.2.0 e1d6a67b021e fe000a9cec38 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:node-exporter.a smithi043 *:9100 running (4m) 5s ago 4m 15.7M - 1dbe0e931976 695e77613650 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:node-exporter.b smithi181 *:9100 running (4m) 14s ago 4m 14.4M - 1dbe0e931976 dbf7b3ccb3e2 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 smithi043 running (8m) 5s ago 8m 49.2M 4096M 17.2.0 e1d6a67b021e 369dd317255b 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 smithi043 running (8m) 5s ago 8m 49.2M 4096M 17.2.0 e1d6a67b021e 7924546d3d9f 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 smithi043 running (7m) 5s ago 7m 45.7M 4096M 17.2.0 e1d6a67b021e c7e036ddbc42 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 smithi043 running (7m) 5s ago 7m 50.8M 4096M 17.2.0 e1d6a67b021e 5632c3f6a0a3 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 smithi181 running (7m) 14s ago 7m 48.8M 4096M 17.2.0 e1d6a67b021e 383cc066a124 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 smithi181 running (6m) 14s ago 6m 47.1M 4096M 17.2.0 e1d6a67b021e bf96e37f9fad 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 smithi181 running (6m) 14s ago 6m 44.9M 4096M 17.2.0 e1d6a67b021e 8d9702f93d88 2024-05-03T10:29:55.488 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 smithi181 running (5m) 14s ago 5m 51.1M 4096M 17.2.0 e1d6a67b021e 151fde494c40 2024-05-03T10:29:55.489 INFO:teuthology.orchestra.run.smithi043.stdout:prometheus.a smithi181 *:9095 running (3m) 14s ago 5m 64.5M - 514e6a882f6e 906505cdab90 2024-05-03T10:29:55.489 INFO:teuthology.orchestra.run.smithi043.stdout:rgw.foo.smithi043.vtftcc smithi043 *:8000 running (3m) 5s ago 3m 68.8M - 17.2.0 e1d6a67b021e 5bd07849f372 2024-05-03T10:29:55.489 INFO:teuthology.orchestra.run.smithi043.stdout:rgw.foo.smithi181.kgwjoi smithi181 *:8000 running (3m) 14s ago 3m 69.0M - 17.2.0 e1d6a67b021e 731a5acbab24 2024-05-03T10:29:56.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:56 smithi043 ceph-mon[22404]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:56.293 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:56 smithi043 ceph-mon[22404]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:29:56.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:56 smithi043 ceph-mon[27243]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:56.293 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:56 smithi043 ceph-mon[27243]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:29:56.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:56 smithi181 ceph-mon[23393]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:56.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:56 smithi181 ceph-mon[23393]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:29:56.370 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph versions' 2024-05-03T10:29:58.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:57 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:58.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:57 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:58.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:57 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:29:58.750 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:58 smithi043 ceph-mon[22404]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:58.750 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:58 smithi043 ceph-mon[27243]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:59.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:58 smithi181 ceph-mon[23393]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:29:59.483 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": { 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": { 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": { 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": {}, 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": { 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:29:59.484 INFO:teuthology.orchestra.run.smithi043.stdout: "overall": { 2024-05-03T10:29:59.485 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-05-03T10:29:59.485 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:29:59.485 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-03T10:29:59.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:29:59 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/2654755419' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-03T10:29:59.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:29:59 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/2654755419' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-03T10:29:59.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:29:59 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/2654755419' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-03T10:29:59.870 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:29:59 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:29:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:30:00.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph -s' 2024-05-03T10:30:00.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:00 smithi043 ceph-mon[27243]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:00.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:00 smithi043 ceph-mon[27243]: overall HEALTH_OK 2024-05-03T10:30:00.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:00 smithi043 ceph-mon[22404]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:00.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:00 smithi043 ceph-mon[22404]: overall HEALTH_OK 2024-05-03T10:30:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:00 smithi181 ceph-mon[23393]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:00 smithi181 ceph-mon[23393]: overall HEALTH_OK 2024-05-03T10:30:01.541 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:01 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:30:01] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:30:02.368 INFO:teuthology.orchestra.run.smithi043.stdout: cluster: 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: id: 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: health: HEALTH_OK 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: services: 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: mon: 3 daemons, quorum a,c,b (age 9m) 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: mgr: y(active, since 5m), standbys: x 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 6m) 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: data: 2024-05-03T10:30:02.369 INFO:teuthology.orchestra.run.smithi043.stdout: pools: 6 pools, 161 pgs 2024-05-03T10:30:02.370 INFO:teuthology.orchestra.run.smithi043.stdout: objects: 209 objects, 585 KiB 2024-05-03T10:30:02.370 INFO:teuthology.orchestra.run.smithi043.stdout: usage: 68 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:30:02.370 INFO:teuthology.orchestra.run.smithi043.stdout: pgs: 161 active+clean 2024-05-03T10:30:02.370 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:30:02.370 INFO:teuthology.orchestra.run.smithi043.stdout: io: 2024-05-03T10:30:02.370 INFO:teuthology.orchestra.run.smithi043.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-05-03T10:30:02.370 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:30:02.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:02 smithi043 ceph-mon[22404]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:02.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:02.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:02.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:02 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3471317582' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-03T10:30:02.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:02 smithi043 ceph-mon[27243]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:02.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:02.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:02.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:02 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3471317582' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-03T10:30:02.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:02 smithi181 ceph-mon[23393]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:02.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:02.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:02.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:02 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3471317582' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-03T10:30:03.277 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph orch ls' 2024-05-03T10:30:03.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:03 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:03] "GET /metrics HTTP/1.1" 200 207922 "" "Prometheus/2.33.4" 2024-05-03T10:30:04.770 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:04 smithi043 ceph-mon[27243]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:04.770 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:04 smithi043 ceph-mon[22404]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:04.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:04 smithi181 ceph-mon[23393]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:05.455 INFO:teuthology.orchestra.run.smithi043.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-03T10:30:05.455 INFO:teuthology.orchestra.run.smithi043.stdout:agent 2/2 15s ago 5m * 2024-05-03T10:30:05.455 INFO:teuthology.orchestra.run.smithi043.stdout:alertmanager ?:9093,9094 1/1 15s ago 5m smithi043=a;count:1 2024-05-03T10:30:05.455 INFO:teuthology.orchestra.run.smithi043.stdout:grafana ?:3000 1/1 4s ago 5m smithi181=a;count:1 2024-05-03T10:30:05.456 INFO:teuthology.orchestra.run.smithi043.stdout:iscsi.foo 1/1 15s ago 3m count:1 2024-05-03T10:30:05.456 INFO:teuthology.orchestra.run.smithi043.stdout:mgr 2/2 15s ago 9m smithi043=y;smithi181=x;count:2 2024-05-03T10:30:05.456 INFO:teuthology.orchestra.run.smithi043.stdout:mon 3/3 15s ago 9m smithi043:172.21.15.43=a;smithi043:[v2:172.21.15.43:3301,v1:172.21.15.43:6790]=c;smithi181:172.21.15.181=b;count:3 2024-05-03T10:30:05.456 INFO:teuthology.orchestra.run.smithi043.stdout:node-exporter ?:9100 2/2 15s ago 5m smithi043=a;smithi181=b;count:2 2024-05-03T10:30:05.456 INFO:teuthology.orchestra.run.smithi043.stdout:osd 8 15s ago - 2024-05-03T10:30:05.456 INFO:teuthology.orchestra.run.smithi043.stdout:prometheus ?:9095 1/1 4s ago 5m smithi181=a;count:1 2024-05-03T10:30:05.456 INFO:teuthology.orchestra.run.smithi043.stdout:rgw.foo ?:8000 2/2 15s ago 3m count:2 2024-05-03T10:30:06.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-05-03T10:30:06.671 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:06 smithi181 ceph-mon[23393]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:06.672 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:06 smithi181 ceph-mon[23393]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:30:06.715 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:06 smithi043 ceph-mon[27243]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:06.715 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:06 smithi043 ceph-mon[27243]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:30:06.716 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:06 smithi043 ceph-mon[22404]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:06.716 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:06 smithi043 ceph-mon[22404]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:30:07.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:07 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:07.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:07 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:08.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:07 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:08.591 INFO:teuthology.orchestra.run.smithi043.stdout:Scheduled to redeploy mgr.x on host 'smithi181' 2024-05-03T10:30:08.919 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:08 smithi043 ceph-mon[27243]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:08.919 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:08 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/1401262138' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T10:30:08.919 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:08 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:08.919 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:08 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:30:08.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:08 smithi043 ceph-mon[22404]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:08.919 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:08 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/1401262138' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T10:30:08.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:08 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:08.920 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:08 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:30:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:08 smithi181 ceph-mon[23393]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:08 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/1401262138' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T10:30:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:08 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:08 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:30:09.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph orch ps --refresh' 2024-05-03T10:30:09.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:30:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:30:09.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:09 smithi181 ceph-mon[23393]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:f8e6d2d721bc96a93915917c59bc3d287d57263d", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:30:09.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:09 smithi181 ceph-mon[23393]: Schedule redeploy daemon mgr.x 2024-05-03T10:30:09.949 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:09 smithi043 ceph-mon[22404]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:f8e6d2d721bc96a93915917c59bc3d287d57263d", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:30:09.949 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:09 smithi043 ceph-mon[22404]: Schedule redeploy daemon mgr.x 2024-05-03T10:30:09.949 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:09 smithi043 ceph-mon[27243]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:f8e6d2d721bc96a93915917c59bc3d287d57263d", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:30:09.950 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:09 smithi043 ceph-mon[27243]: Schedule redeploy daemon mgr.x 2024-05-03T10:30:09.950 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:09 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:30:09] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:30:10.907 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:10 smithi043 ceph-mon[27243]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:10.907 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:10.907 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:10.907 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:10 smithi043 ceph-mon[22404]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:10.908 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:10.908 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:10 smithi181 ceph-mon[23393]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:11.372 INFO:teuthology.orchestra.run.smithi043.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-03T10:30:11.372 INFO:teuthology.orchestra.run.smithi043.stdout:agent.smithi043 smithi043 running 1s ago 10m - - 2024-05-03T10:30:11.372 INFO:teuthology.orchestra.run.smithi043.stdout:agent.smithi181 smithi181 running 9s ago 9m - - 2024-05-03T10:30:11.372 INFO:teuthology.orchestra.run.smithi043.stdout:alertmanager.a smithi043 *:9093,9094 running (4m) 1s ago 4m 28.3M - ba2b418f427c 43fbcd25b859 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:grafana.a smithi181 *:3000 running (4m) 9s ago 4m 54.7M - 8.3.5 dad864ee21e9 15a635565ef8 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:iscsi.foo.smithi043.lxwugw smithi043 running (3m) 1s ago 3m 47.0M - 3.5 e1d6a67b021e 89b7cc11efa9 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.x smithi181 *:8443 running (9m) 9s ago 9m 412M - 17.2.0 e1d6a67b021e f42a19b78d32 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.y smithi043 *:9283 running (10m) 1s ago 10m 465M - 17.2.0 e1d6a67b021e e925f3a624c5 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a smithi043 running (11m) 1s ago 11m 50.1M 2048M 17.2.0 e1d6a67b021e 92837a4d1736 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b smithi181 running (9m) 9s ago 9m 38.1M 2048M 17.2.0 e1d6a67b021e 5e27fbb56907 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c smithi043 running (9m) 1s ago 9m 37.4M 2048M 17.2.0 e1d6a67b021e fe000a9cec38 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:node-exporter.a smithi043 *:9100 running (5m) 1s ago 5m 16.3M - 1dbe0e931976 695e77613650 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:node-exporter.b smithi181 *:9100 running (5m) 9s ago 5m 14.3M - 1dbe0e931976 dbf7b3ccb3e2 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 smithi043 running (8m) 1s ago 8m 49.3M 4096M 17.2.0 e1d6a67b021e 369dd317255b 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 smithi043 running (8m) 1s ago 8m 49.4M 4096M 17.2.0 e1d6a67b021e 7924546d3d9f 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 smithi043 running (8m) 1s ago 8m 45.9M 4096M 17.2.0 e1d6a67b021e c7e036ddbc42 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 smithi043 running (7m) 1s ago 7m 51.0M 4096M 17.2.0 e1d6a67b021e 5632c3f6a0a3 2024-05-03T10:30:11.373 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 smithi181 running (7m) 9s ago 7m 49.0M 4096M 17.2.0 e1d6a67b021e 383cc066a124 2024-05-03T10:30:11.374 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 smithi181 running (6m) 9s ago 6m 47.3M 4096M 17.2.0 e1d6a67b021e bf96e37f9fad 2024-05-03T10:30:11.374 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 smithi181 running (6m) 9s ago 6m 45.0M 4096M 17.2.0 e1d6a67b021e 8d9702f93d88 2024-05-03T10:30:11.374 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 smithi181 running (6m) 9s ago 6m 51.3M 4096M 17.2.0 e1d6a67b021e 151fde494c40 2024-05-03T10:30:11.374 INFO:teuthology.orchestra.run.smithi043.stdout:prometheus.a smithi181 *:9095 running (4m) 9s ago 5m 64.5M - 514e6a882f6e 906505cdab90 2024-05-03T10:30:11.374 INFO:teuthology.orchestra.run.smithi043.stdout:rgw.foo.smithi043.vtftcc smithi043 *:8000 running (3m) 1s ago 3m 68.9M - 17.2.0 e1d6a67b021e 5bd07849f372 2024-05-03T10:30:11.374 INFO:teuthology.orchestra.run.smithi043.stdout:rgw.foo.smithi181.kgwjoi smithi181 *:8000 running (3m) 9s ago 3m 69.1M - 17.2.0 e1d6a67b021e 731a5acbab24 2024-05-03T10:30:12.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:11 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:30:12.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:11 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:30:12.077 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'sleep 180' 2024-05-03T10:30:12.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:11 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:30:13.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:12 smithi043 ceph-mon[27243]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:13.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:12 smithi043 ceph-mon[27243]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:30:13.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:12 smithi043 ceph-mon[22404]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:13.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:12 smithi043 ceph-mon[22404]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:30:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:12 smithi181 ceph-mon[23393]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:12 smithi181 ceph-mon[23393]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:30:13.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:13] "GET /metrics HTTP/1.1" 200 207916 "" "Prometheus/2.33.4" 2024-05-03T10:30:15.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:14 smithi043 ceph-mon[22404]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:15.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:14 smithi043 ceph-mon[27243]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:14 smithi181 ceph-mon[23393]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:17.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:16 smithi043 ceph-mon[22404]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:17.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:16 smithi043 ceph-mon[27243]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:17.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:16 smithi181 ceph-mon[23393]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:17.756 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:17 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:17.756 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:17 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:17.957 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:17 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:19.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:18 smithi043 ceph-mon[22404]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:19.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:18 smithi043 ceph-mon[27243]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:19.051 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:18 smithi181 ceph-mon[23393]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:19.838 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:30:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:30:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:20 smithi043 ceph-mon[22404]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:21.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:20 smithi043 ceph-mon[27243]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:20 smithi181 ceph-mon[23393]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:21.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:21 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:30:21] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:30:22.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:22 smithi043 ceph-mon[22404]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:22.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:22.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:22.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:22 smithi043 ceph-mon[27243]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:22.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:22.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:22.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:22 smithi181 ceph-mon[23393]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:22.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:22.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:23.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:23] "GET /metrics HTTP/1.1" 200 207916 "" "Prometheus/2.33.4" 2024-05-03T10:30:24.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:24 smithi043 ceph-mon[22404]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:24.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:24 smithi043 ceph-mon[27243]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:24.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:24 smithi181 ceph-mon[23393]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:26.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:26 smithi043 ceph-mon[22404]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:26.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:26 smithi043 ceph-mon[27243]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:26.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:26 smithi181 ceph-mon[23393]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:27 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:30:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:30:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:27 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:30:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:30:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:27 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:30:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:30:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:28 smithi043 ceph-mon[22404]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:29.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:28 smithi043 ceph-mon[27243]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:28 smithi181 ceph-mon[23393]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:29.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:30:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:30:30.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:29 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:30:29] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:30:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:30 smithi043 ceph-mon[22404]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:30 smithi043 ceph-mon[27243]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:30 smithi181 ceph-mon[23393]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:33.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:32 smithi043 ceph-mon[22404]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:33.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:32 smithi043 ceph-mon[27243]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:33.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:32 smithi181 ceph-mon[23393]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:33.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:33 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:33] "GET /metrics HTTP/1.1" 200 207921 "" "Prometheus/2.33.4" 2024-05-03T10:30:35.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:34 smithi043 ceph-mon[22404]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:35.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:34 smithi043 ceph-mon[27243]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:35.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:34 smithi181 ceph-mon[23393]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:37.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:36 smithi043 ceph-mon[22404]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:37.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:36 smithi043 ceph-mon[27243]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:37.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:36 smithi181 ceph-mon[23393]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:37.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:37 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:37.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:37 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:38.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:37 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:38.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:38 smithi181 ceph-mon[23393]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:39.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:38 smithi043 ceph-mon[22404]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:39.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:38 smithi043 ceph-mon[27243]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:39.778 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:30:39 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:30:41.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:40 smithi043 ceph-mon[22404]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:41.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:40 smithi043 ceph-mon[27243]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:41.099 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:40 smithi181 ceph-mon[23393]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:42.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:41 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:30:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:30:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:42 smithi181 ceph-mon[23393]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:43.044 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:42 smithi043 ceph-mon[22404]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:43.044 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:43.045 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:43.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:42 smithi043 ceph-mon[27243]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:43.048 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:43.048 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:43.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:43] "GET /metrics HTTP/1.1" 200 207924 "" "Prometheus/2.33.4" 2024-05-03T10:30:44.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:44 smithi181 ceph-mon[23393]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:44 smithi043 ceph-mon[22404]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:45.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:44 smithi043 ceph-mon[27243]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:46.694 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:46 smithi043 ceph-mon[22404]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:46.694 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:46 smithi043 ceph-mon[27243]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:46.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:46 smithi181 ceph-mon[23393]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:47.956 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:47 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:47.957 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:47 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:47 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:49.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:48 smithi043 ceph-mon[22404]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:49.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:48 smithi043 ceph-mon[27243]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:48 smithi181 ceph-mon[23393]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:49.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:30:49 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:30:50.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:50 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:30:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:30:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:50 smithi043 ceph-mon[22404]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:51.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:51.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:50 smithi043 ceph-mon[27243]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:50 smithi181 ceph-mon[23393]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:30:53.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:52 smithi043 ceph-mon[22404]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:53.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:52 smithi043 ceph-mon[27243]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:52 smithi181 ceph-mon[23393]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:53.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:30:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:53] "GET /metrics HTTP/1.1" 200 207924 "" "Prometheus/2.33.4" 2024-05-03T10:30:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:54 smithi043 ceph-mon[22404]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:55.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:54 smithi043 ceph-mon[27243]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:54 smithi181 ceph-mon[23393]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:30:56.972 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:56 smithi043 ceph-mon[22404]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:56.973 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:56 smithi043 ceph-mon[27243]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:56 smithi181 ceph-mon[23393]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:57.858 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:57 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:57.859 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:57 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:58.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:57 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:30:59.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:30:58 smithi043 ceph-mon[22404]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:59.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:30:58 smithi043 ceph-mon[27243]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:59.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:30:58 smithi181 ceph-mon[23393]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:30:59.823 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:30:59 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:30:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:31:00.991 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:00 smithi181 ceph-mon[23393]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:01.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:00 smithi043 ceph-mon[22404]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:01.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:00 smithi043 ceph-mon[27243]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:02.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:01 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:31:01] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:31:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:02 smithi043 ceph-mon[22404]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:03.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:02 smithi043 ceph-mon[27243]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:02 smithi181 ceph-mon[23393]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:03.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:03 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:03] "GET /metrics HTTP/1.1" 200 207909 "" "Prometheus/2.33.4" 2024-05-03T10:31:05.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:04 smithi043 ceph-mon[27243]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:05.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:04 smithi043 ceph-mon[22404]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:05.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:04 smithi181 ceph-mon[23393]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:07.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:06 smithi043 ceph-mon[22404]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:07.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:06 smithi043 ceph-mon[27243]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:07.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:06 smithi181 ceph-mon[23393]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:08.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:07 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:08.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:07 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:08.069 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:07 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:09.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:08 smithi043 ceph-mon[27243]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:09.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:08 smithi043 ceph-mon[22404]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:08 smithi181 ceph-mon[23393]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:09.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:31:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:31:10.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:10 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:31:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:31:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:10 smithi043 ceph-mon[22404]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:11.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:10 smithi043 ceph-mon[27243]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:11.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:11.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:10 smithi181 ceph-mon[23393]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:13.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:12 smithi043 ceph-mon[22404]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:13.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:12 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:31:13.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:12 smithi043 ceph-mon[27243]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:13.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:12 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:31:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:12 smithi181 ceph-mon[23393]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:12 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:31:13.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:13] "GET /metrics HTTP/1.1" 200 207918 "" "Prometheus/2.33.4" 2024-05-03T10:31:15.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:14 smithi043 ceph-mon[22404]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:15.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:14 smithi043 ceph-mon[27243]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:14 smithi181 ceph-mon[23393]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:17.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:16 smithi043 ceph-mon[22404]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:17.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:16 smithi043 ceph-mon[27243]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:17.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:16 smithi181 ceph-mon[23393]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:17.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:17 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:18.041 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:17 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:17 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:19.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:18 smithi043 ceph-mon[22404]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:19.047 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:18 smithi043 ceph-mon[27243]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:18 smithi181 ceph-mon[23393]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:19.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:31:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:31:20.824 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:20 smithi181 ceph-mon[23393]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:20 smithi043 ceph-mon[22404]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:21.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:20 smithi043 ceph-mon[27243]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:22.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:21 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:31:21] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:31:23.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:22 smithi043 ceph-mon[27243]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:23.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:23.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:23.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:22 smithi043 ceph-mon[22404]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:23.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:23.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:22 smithi181 ceph-mon[23393]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:23.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:23] "GET /metrics HTTP/1.1" 200 207918 "" "Prometheus/2.33.4" 2024-05-03T10:31:25.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:24 smithi043 ceph-mon[22404]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:25.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:24 smithi043 ceph-mon[27243]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:25.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:24 smithi181 ceph-mon[23393]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:27.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:26 smithi043 ceph-mon[22404]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:27.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:26 smithi043 ceph-mon[27243]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:27.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:26 smithi181 ceph-mon[23393]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:27 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:31:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:31:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:27 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:31:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:31:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:27 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:31:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:31:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:28 smithi043 ceph-mon[22404]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:29.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:28 smithi043 ceph-mon[27243]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:28 smithi181 ceph-mon[23393]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:29.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:31:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:31:30.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:30 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:31:30] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:31:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:30 smithi043 ceph-mon[22404]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:30 smithi043 ceph-mon[27243]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:30 smithi181 ceph-mon[23393]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:33.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:32 smithi043 ceph-mon[22404]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:33.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:32 smithi043 ceph-mon[27243]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:33.044 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:32 smithi181 ceph-mon[23393]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:33.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:33 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:33] "GET /metrics HTTP/1.1" 200 207923 "" "Prometheus/2.33.4" 2024-05-03T10:31:35.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:34 smithi043 ceph-mon[27243]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:35.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:34 smithi043 ceph-mon[22404]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:35.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:34 smithi181 ceph-mon[23393]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:37.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:36 smithi043 ceph-mon[22404]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:37.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:36 smithi043 ceph-mon[27243]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:37.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:36 smithi181 ceph-mon[23393]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:38.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:37 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:38.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:37 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:38.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:37 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:39.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:38 smithi043 ceph-mon[22404]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:39.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:38 smithi043 ceph-mon[27243]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:39.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:38 smithi181 ceph-mon[23393]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:39.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:31:39 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:31:41.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:40 smithi043 ceph-mon[22404]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:41.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:40 smithi043 ceph-mon[27243]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:41.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:40 smithi181 ceph-mon[23393]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:42.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:41 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:31:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:31:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:42 smithi181 ceph-mon[23393]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:42.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:43.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:42 smithi043 ceph-mon[22404]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:43.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:43.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:43.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:42 smithi043 ceph-mon[27243]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:43.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:43.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:43.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:43] "GET /metrics HTTP/1.1" 200 207916 "" "Prometheus/2.33.4" 2024-05-03T10:31:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:44 smithi043 ceph-mon[22404]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:45.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:44 smithi043 ceph-mon[27243]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:45.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:44 smithi181 ceph-mon[23393]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:47.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:46 smithi043 ceph-mon[22404]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:47.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:46 smithi043 ceph-mon[27243]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:47.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:46 smithi181 ceph-mon[23393]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:48.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:47 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:48.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:47 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:47 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:49.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:48 smithi043 ceph-mon[27243]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:49.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:48 smithi043 ceph-mon[22404]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:48 smithi181 ceph-mon[23393]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:49.784 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:31:49 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:31:50.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:49 smithi043 ceph-mon[22404]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:50.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:49 smithi043 ceph-mon[27243]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:50.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:49 smithi181 ceph-mon[23393]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:50.792 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:50 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:31:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:31:51.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:51 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:51.619 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:51 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:51.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:51 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:51.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:51 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:51.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:51 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:51.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:51 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:31:53.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:52 smithi043 ceph-mon[22404]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:53.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:52 smithi043 ceph-mon[27243]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:52 smithi181 ceph-mon[23393]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:53.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:31:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:53] "GET /metrics HTTP/1.1" 200 207916 "" "Prometheus/2.33.4" 2024-05-03T10:31:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:54 smithi043 ceph-mon[22404]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:55.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:54 smithi043 ceph-mon[27243]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:54 smithi181 ceph-mon[23393]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:31:57.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:56 smithi043 ceph-mon[22404]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:57.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:56 smithi043 ceph-mon[27243]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:56 smithi181 ceph-mon[23393]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:57.811 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:57 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:57.812 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:57 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:58.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:57 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:31:58.922 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:31:58 smithi043 ceph-mon[27243]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:58.922 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:31:58 smithi043 ceph-mon[22404]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:59.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:31:58 smithi181 ceph-mon[23393]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:31:59.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:31:59 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:31:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:32:01.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:00 smithi043 ceph-mon[27243]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:01.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:00 smithi043 ceph-mon[22404]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:00 smithi181 ceph-mon[23393]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:02.292 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:02 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:32:02] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:32:03.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:02 smithi181 ceph-mon[23393]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:03.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:03.035 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:02 smithi043 ceph-mon[22404]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:03.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:02 smithi043 ceph-mon[27243]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:03.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:03.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:03.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:03 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:03] "GET /metrics HTTP/1.1" 200 207920 "" "Prometheus/2.33.4" 2024-05-03T10:32:04.779 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:04 smithi181 ceph-mon[23393]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:05.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:04 smithi043 ceph-mon[22404]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:05.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:04 smithi043 ceph-mon[27243]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:07.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:06 smithi043 ceph-mon[22404]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:07.046 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:06 smithi043 ceph-mon[27243]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:07.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:06 smithi181 ceph-mon[23393]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:08.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:07 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:08.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:07 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:08.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:07 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:09.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:08 smithi043 ceph-mon[22404]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:09.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:08 smithi043 ceph-mon[27243]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:08 smithi181 ceph-mon[23393]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:09.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:32:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:32:10.747 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:10 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:32:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:32:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:10 smithi043 ceph-mon[22404]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:11.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:10 smithi043 ceph-mon[27243]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:11.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:11.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:10 smithi181 ceph-mon[23393]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:13.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:12 smithi043 ceph-mon[27243]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:13.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:12 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:32:13.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:12 smithi043 ceph-mon[22404]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:13.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:12 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:32:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:12 smithi181 ceph-mon[23393]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:12 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:32:13.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:13] "GET /metrics HTTP/1.1" 200 207922 "" "Prometheus/2.33.4" 2024-05-03T10:32:15.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:14 smithi043 ceph-mon[22404]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:15.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:14 smithi043 ceph-mon[27243]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:14 smithi181 ceph-mon[23393]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:17.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:16 smithi043 ceph-mon[22404]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:17.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:16 smithi043 ceph-mon[27243]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:17.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:16 smithi181 ceph-mon[23393]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:17.971 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:17 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:17.971 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:17 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:18.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:17 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:19.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:18 smithi043 ceph-mon[22404]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:19.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:18 smithi043 ceph-mon[27243]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:18 smithi181 ceph-mon[23393]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:19.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:32:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:32:21.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:20 smithi043 ceph-mon[22404]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:21.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:20 smithi043 ceph-mon[27243]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:21.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:20 smithi181 ceph-mon[23393]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:22.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:22 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:32:22] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:32:23.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:22 smithi043 ceph-mon[22404]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:23.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:23.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:23.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:22 smithi043 ceph-mon[27243]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:23.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:23.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:22 smithi181 ceph-mon[23393]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:23.120 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:23.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:23] "GET /metrics HTTP/1.1" 200 207922 "" "Prometheus/2.33.4" 2024-05-03T10:32:25.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:24 smithi043 ceph-mon[22404]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:25.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:24 smithi043 ceph-mon[27243]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:25.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:24 smithi181 ceph-mon[23393]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:27.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:26 smithi043 ceph-mon[22404]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:27.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:26 smithi043 ceph-mon[27243]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:27.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:26 smithi181 ceph-mon[23393]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:27 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:32:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:32:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:27 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:32:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:32:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:27 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:32:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:32:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:28 smithi043 ceph-mon[22404]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:29.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:28 smithi043 ceph-mon[27243]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:28 smithi181 ceph-mon[23393]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:29.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:32:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:32:31.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:30 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:32:30] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:32:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:30 smithi043 ceph-mon[27243]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:31.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:30 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:31.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:30 smithi043 ceph-mon[22404]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:31.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:31.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:30 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:30 smithi181 ceph-mon[23393]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:31.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:30 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:33.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:32 smithi043 ceph-mon[22404]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:33.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:32 smithi043 ceph-mon[27243]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:33.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:32 smithi181 ceph-mon[23393]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:33.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:33 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:33] "GET /metrics HTTP/1.1" 200 207918 "" "Prometheus/2.33.4" 2024-05-03T10:32:34.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:33 smithi181 ceph-mon[23393]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:34.292 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:33 smithi043 ceph-mon[22404]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:34.292 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:33 smithi043 ceph-mon[27243]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:36.369 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:36 smithi181 ceph-mon[23393]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:36.507 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:36 smithi043 ceph-mon[22404]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:36.508 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:36 smithi043 ceph-mon[27243]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:37.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:37 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:37.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:37 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:38.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:37 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:38.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:38 smithi043 ceph-mon[22404]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:38.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:38 smithi043 ceph-mon[27243]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:39.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:38 smithi181 ceph-mon[23393]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:39.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:32:39 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:32:41.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:40 smithi043 ceph-mon[22404]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:41.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:40 smithi043 ceph-mon[27243]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:41.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:40 smithi181 ceph-mon[23393]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:42.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:42 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:32:42] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:32:43.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:42 smithi043 ceph-mon[22404]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:43.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:43.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:42 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:43.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:42 smithi043 ceph-mon[27243]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:43.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:43.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:42 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:43.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:42 smithi181 ceph-mon[23393]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:43.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:43.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:42 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:43.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:43 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:43] "GET /metrics HTTP/1.1" 200 207924 "" "Prometheus/2.33.4" 2024-05-03T10:32:44.837 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:44 smithi181 ceph-mon[23393]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:45.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:44 smithi043 ceph-mon[22404]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:45.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:44 smithi043 ceph-mon[27243]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:46.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:46 smithi043 ceph-mon[22404]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:46.793 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:46 smithi043 ceph-mon[27243]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:46.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:46 smithi181 ceph-mon[23393]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:48.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:47 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:48.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:47 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:48.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:47 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:49.000 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:48 smithi043 ceph-mon[27243]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:49.000 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:48 smithi043 ceph-mon[22404]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:49.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:48 smithi181 ceph-mon[23393]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:49.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:32:49 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:32:51.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:50 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:32:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:32:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:50 smithi043 ceph-mon[27243]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:51.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:50 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:51.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:50 smithi043 ceph-mon[22404]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:51.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:51.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:50 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:50 smithi181 ceph-mon[23393]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:51.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:50 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:32:53.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:52 smithi043 ceph-mon[27243]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:53.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:52 smithi043 ceph-mon[22404]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:53.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:52 smithi181 ceph-mon[23393]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:53.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:32:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:53] "GET /metrics HTTP/1.1" 200 207924 "" "Prometheus/2.33.4" 2024-05-03T10:32:55.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:54 smithi043 ceph-mon[22404]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:55.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:54 smithi043 ceph-mon[27243]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:55.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:54 smithi181 ceph-mon[23393]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:32:56.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:56 smithi043 ceph-mon[22404]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:56.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:56 smithi043 ceph-mon[27243]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:56 smithi181 ceph-mon[23393]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:57.869 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:57 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:58.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:57 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:58.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:57 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:32:59.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:32:58 smithi043 ceph-mon[22404]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:59.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:32:58 smithi043 ceph-mon[27243]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:59.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:32:58 smithi181 ceph-mon[23393]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:32:59.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:32:59 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:32:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:33:01.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:00 smithi043 ceph-mon[22404]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:01.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:00 smithi043 ceph-mon[27243]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:01.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:00 smithi181 ceph-mon[23393]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:02.766 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:02 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:33:02] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:33:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:02 smithi043 ceph-mon[22404]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:03.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:02 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:02 smithi043 ceph-mon[27243]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:03.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:02 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:02 smithi181 ceph-mon[23393]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:03.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:02 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:03.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:03 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:33:03] "GET /metrics HTTP/1.1" 200 207921 "" "Prometheus/2.33.4" 2024-05-03T10:33:05.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:04 smithi043 ceph-mon[22404]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:05.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:04 smithi043 ceph-mon[27243]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:05.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:04 smithi181 ceph-mon[23393]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:06.792 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:06 smithi043 ceph-mon[22404]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:06.792 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:06 smithi043 ceph-mon[27243]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:07.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:06 smithi181 ceph-mon[23393]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:08.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:07 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:33:08.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:07 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:33:08.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:07 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:33:09.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:08 smithi043 ceph-mon[22404]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:09.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:08 smithi043 ceph-mon[27243]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:09.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:08 smithi181 ceph-mon[23393]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:09.869 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:33:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:33:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:33:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:10 smithi043 ceph-mon[22404]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:11.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:10 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:11.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:10 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.43 - - [03/May/2024:10:33:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:33:11.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:10 smithi043 ceph-mon[27243]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:11.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:11.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:10 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:10 smithi181 ceph-mon[23393]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:11.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:10 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:13.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:12 smithi043 ceph-mon[22404]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:13.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:12 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:33:13.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:12 smithi043 ceph-mon[27243]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:13.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:12 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:33:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:12 smithi181 ceph-mon[23393]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:13.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:12 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T10:33:13.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:13 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:33:13] "GET /metrics HTTP/1.1" 200 207923 "" "Prometheus/2.33.4" 2024-05-03T10:33:14.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph orch ps' 2024-05-03T10:33:14.967 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:14 smithi043 ceph-mon[22404]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:14.967 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:14 smithi043 ceph-mon[27243]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:15.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:14 smithi181 ceph-mon[23393]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:16.630 INFO:teuthology.orchestra.run.smithi043.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:agent.smithi043 smithi043 running 5s ago 13m - - 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:agent.smithi181 smithi181 running 14s ago 13m - - 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:alertmanager.a smithi043 *:9093,9094 running (7m) 5s ago 7m 25.6M - ba2b418f427c 43fbcd25b859 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:grafana.a smithi181 *:3000 running (7m) 14s ago 7m 54.4M - 8.3.5 dad864ee21e9 15a635565ef8 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:iscsi.foo.smithi043.lxwugw smithi043 running (6m) 5s ago 6m 52.4M - 3.5 e1d6a67b021e 89b7cc11efa9 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.x smithi181 *:8443 running (12m) 14s ago 12m 413M - 17.2.0 e1d6a67b021e f42a19b78d32 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:mgr.y smithi043 *:9283 running (14m) 5s ago 14m 470M - 17.2.0 e1d6a67b021e e925f3a624c5 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:mon.a smithi043 running (14m) 5s ago 14m 52.5M 2048M 17.2.0 e1d6a67b021e 92837a4d1736 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:mon.b smithi181 running (12m) 14s ago 12m 42.5M 2048M 17.2.0 e1d6a67b021e 5e27fbb56907 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:mon.c smithi043 running (12m) 5s ago 12m 42.5M 2048M 17.2.0 e1d6a67b021e fe000a9cec38 2024-05-03T10:33:16.631 INFO:teuthology.orchestra.run.smithi043.stdout:node-exporter.a smithi043 *:9100 running (8m) 5s ago 8m 16.0M - 1dbe0e931976 695e77613650 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:node-exporter.b smithi181 *:9100 running (8m) 14s ago 8m 16.3M - 1dbe0e931976 dbf7b3ccb3e2 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:osd.0 smithi043 running (12m) 5s ago 12m 50.8M 4096M 17.2.0 e1d6a67b021e 369dd317255b 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:osd.1 smithi043 running (11m) 5s ago 11m 50.7M 4096M 17.2.0 e1d6a67b021e 7924546d3d9f 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:osd.2 smithi043 running (11m) 5s ago 11m 47.1M 4096M 17.2.0 e1d6a67b021e c7e036ddbc42 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:osd.3 smithi043 running (10m) 5s ago 10m 52.5M 4096M 17.2.0 e1d6a67b021e 5632c3f6a0a3 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:osd.4 smithi181 running (10m) 14s ago 10m 50.3M 4096M 17.2.0 e1d6a67b021e 383cc066a124 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:osd.5 smithi181 running (10m) 14s ago 10m 48.6M 4096M 17.2.0 e1d6a67b021e bf96e37f9fad 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:osd.6 smithi181 running (9m) 14s ago 9m 46.2M 4096M 17.2.0 e1d6a67b021e 8d9702f93d88 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:osd.7 smithi181 running (9m) 14s ago 9m 52.5M 4096M 17.2.0 e1d6a67b021e 151fde494c40 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:prometheus.a smithi181 *:9095 running (7m) 14s ago 8m 66.8M - 514e6a882f6e 906505cdab90 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:rgw.foo.smithi043.vtftcc smithi043 *:8000 running (6m) 5s ago 6m 69.7M - 17.2.0 e1d6a67b021e 5bd07849f372 2024-05-03T10:33:16.632 INFO:teuthology.orchestra.run.smithi043.stdout:rgw.foo.smithi181.kgwjoi smithi181 *:8000 running (6m) 14s ago 6m 69.8M - 17.2.0 e1d6a67b021e 731a5acbab24 2024-05-03T10:33:17.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:16 smithi043 ceph-mon[22404]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:17.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:16 smithi043 ceph-mon[27243]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:17.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:16 smithi181 ceph-mon[23393]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:17.231 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph versions' 2024-05-03T10:33:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:17 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:33:18.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:17 smithi043 ceph-mon[22404]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:33:18.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:17 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:33:18.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:17 smithi043 ceph-mon[27243]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:33:18.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:17 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:33:18.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:17 smithi181 ceph-mon[23393]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T10:33:18.917 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:18 smithi043 ceph-mon[27243]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:18.917 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:18 smithi043 ceph-mon[22404]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:19.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:18 smithi181 ceph-mon[23393]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:19.398 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": { 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": { 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": { 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: "mds": {}, 2024-05-03T10:33:19.399 INFO:teuthology.orchestra.run.smithi043.stdout: "rgw": { 2024-05-03T10:33:19.400 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-03T10:33:19.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-05-03T10:33:19.400 INFO:teuthology.orchestra.run.smithi043.stdout: "overall": { 2024-05-03T10:33:19.400 INFO:teuthology.orchestra.run.smithi043.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-05-03T10:33:19.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-05-03T10:33:19.400 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-05-03T10:33:19.801 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:33:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:33:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:33:20.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:19 smithi043 ceph-mon[22404]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:20.072 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:19 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3836176004' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-03T10:33:20.072 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:19 smithi043 ceph-mon[27243]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:20.072 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:19 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3836176004' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-03T10:33:20.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:19 smithi181 ceph-mon[23393]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:20.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:19 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3836176004' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-03T10:33:20.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph -s' 2024-05-03T10:33:22.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:22 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 172.21.15.181 - - [03/May/2024:10:33:22] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-05-03T10:33:23.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:22 smithi043 ceph-mon[22404]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:23.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:23.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:22 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:23.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:22 smithi043 ceph-mon[27243]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:23.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:23.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:22 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:22 smithi181 ceph-mon[23393]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:23.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:22 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' 2024-05-03T10:33:23.350 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:23 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:33:23] "GET /metrics HTTP/1.1" 200 207923 "" "Prometheus/2.33.4" 2024-05-03T10:33:23.463 INFO:teuthology.orchestra.run.smithi043.stdout: cluster: 2024-05-03T10:33:23.463 INFO:teuthology.orchestra.run.smithi043.stdout: id: 79bfd44e-0936-11ef-bc97-c7b262605968 2024-05-03T10:33:23.463 INFO:teuthology.orchestra.run.smithi043.stdout: health: HEALTH_OK 2024-05-03T10:33:23.463 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:33:23.463 INFO:teuthology.orchestra.run.smithi043.stdout: services: 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: mon: 3 daemons, quorum a,c,b (age 12m) 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: mgr: y(active, since 8m), standbys: x 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: osd: 8 osds: 8 up (since 9m), 8 in (since 9m) 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: data: 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: pools: 6 pools, 161 pgs 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: objects: 209 objects, 585 KiB 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: usage: 68 MiB used, 715 GiB / 715 GiB avail 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: pgs: 161 active+clean 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: io: 2024-05-03T10:33:23.464 INFO:teuthology.orchestra.run.smithi043.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-05-03T10:33:23.465 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-05-03T10:33:24.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:23 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3424512870' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-03T10:33:24.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:23 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3424512870' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-03T10:33:24.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:23 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3424512870' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-03T10:33:24.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph health detail' 2024-05-03T10:33:25.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:24 smithi043 ceph-mon[22404]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:25.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:24 smithi043 ceph-mon[27243]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:25.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:24 smithi181 ceph-mon[23393]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-03T10:33:26.182 INFO:teuthology.orchestra.run.smithi043.stdout:HEALTH_OK 2024-05-03T10:33:26.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:26 smithi043 ceph-mon[27243]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:26.777 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:26 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/4137389590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-03T10:33:26.777 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:26 smithi043 ceph-mon[22404]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:26.777 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:26 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/4137389590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-03T10:33:26.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-05-03T10:33:27.054 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:26 smithi181 ceph-mon[23393]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:27.055 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:26 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/4137389590' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-03T10:33:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:27 smithi043 ceph-mon[22404]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:33:28.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:33:28.043 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:27 smithi043 ceph-mon[22404]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:33:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:27 smithi043 ceph-mon[27243]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:33:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:33:28.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:27 smithi043 ceph-mon[27243]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:33:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:27 smithi181 ceph-mon[23393]: from='client.14823 -' entity='client.iscsi.foo.smithi043.lxwugw' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-03T10:33:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-03T10:33:28.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:27 smithi181 ceph-mon[23393]: from='mgr.24328 172.21.15.43:0/1345779571' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-03T10:33:28.989 INFO:teuthology.orchestra.run.smithi043.stdout:false 2024-05-03T10:33:29.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:28 smithi043 ceph-mon[22404]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:29.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:28 smithi043 ceph-mon[27243]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:29.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:28 smithi181 ceph-mon[23393]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-03T10:33:29.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:33:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x[24905]: ::ffff:172.21.15.181 - - [03/May/2024:10:33:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-03T10:33:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T10:33:29.715 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_f8e6d2d721bc96a93915917c59bc3d287d57263d/qa/tasks/cephadm.py", line 1475, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_f8e6d2d721bc96a93915917c59bc3d287d57263d/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi043 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 2\'"\'"\'\'' 2024-05-03T10:33:29.931 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6cb1cf95d415457d8bcd7005e5a5e450 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_f8e6d2d721bc96a93915917c59bc3d287d57263d/qa/tasks/cephadm.py", line 1475, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_f8e6d2d721bc96a93915917c59bc3d287d57263d/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi043 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | length == 2\'"\'"\'\'' 2024-05-03T10:33:29.934 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-03T10:33:29.943 INFO:tasks.cephadm:Teardown begin 2024-05-03T10:33:29.944 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T10:33:29.975 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:29 smithi043 ceph-mon[22404]: from='client.? 172.21.15.43:0/3547617611' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-03T10:33:29.976 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:29 smithi043 ceph-mon[27243]: from='client.? 172.21.15.43:0/3547617611' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-03T10:33:29.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T10:33:30.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:29 smithi181 ceph-mon[23393]: from='client.? 172.21.15.43:0/3547617611' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-03T10:33:30.015 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-03T10:33:30.016 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-03T10:33:30.039 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-03T10:33:30.073 INFO:tasks.cephadm:Stopping all daemons... 2024-05-03T10:33:30.073 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-03T10:33:30.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.a 2024-05-03T10:33:30.289 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:30 smithi043 systemd[1]: Stopping Ceph mon.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:33:30.290 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:30 smithi043 bash[59879]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon.a" found: no such container 2024-05-03T10:33:30.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:30 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a[22400]: 2024-05-03T10:33:30.286+0000 7f0b257bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:33:30.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:30 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a[22400]: 2024-05-03T10:33:30.286+0000 7f0b257bb700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-03T10:33:30.542 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:30 smithi043 podman[59891]: 2024-05-03 10:33:30.449913222 +0000 UTC m=+0.282931574 container died 92837a4d1736fd5e9f8921c7aa0ad49e5a4fa9213ea28f70fadf0b8f67b52791 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, ceph=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.19.8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754) 2024-05-03T10:33:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:30 smithi043 podman[59891]: 2024-05-03 10:33:30.820296657 +0000 UTC m=+0.653315005 container cleanup 92837a4d1736fd5e9f8921c7aa0ad49e5a4fa9213ea28f70fadf0b8f67b52791 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-03T10:33:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:30 smithi043 bash[59891]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-a 2024-05-03T10:33:31.042 INFO:journalctl@ceph.mon.a.smithi043.stdout:May 03 10:33:30 smithi043 bash[59978]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon.a" found: no such container 2024-05-03T10:33:31.079 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.a.service' 2024-05-03T10:33:31.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:33:31.127 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-03T10:33:31.127 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-05-03T10:33:31.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.c 2024-05-03T10:33:31.352 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:31 smithi043 systemd[1]: Stopping Ceph mon.c for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:33:31.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:31 smithi043 bash[60055]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon.c" found: no such container 2024-05-03T10:33:31.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:31 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-c[27227]: 2024-05-03T10:33:31.299+0000 7f4947d4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:33:31.353 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:31 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-c[27227]: 2024-05-03T10:33:31.299+0000 7f4947d4b700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-03T10:33:31.660 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:31 smithi043 podman[60068]: 2024-05-03 10:33:31.351402468 +0000 UTC m=+0.152090360 container died fe000a9cec3812e1a8f997a1e3fb9cc05752c1cb31dbddca10aaa06e8c408912 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-c, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, release=754, GIT_CLEAN=True, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-type=git, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-03T10:33:32.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:31 smithi043 podman[60068]: 2024-05-03 10:33:31.850614675 +0000 UTC m=+0.651302565 container cleanup fe000a9cec3812e1a8f997a1e3fb9cc05752c1cb31dbddca10aaa06e8c408912 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-c, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-type=git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, ceph=True, architecture=x86_64, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.buildah.version=1.19.8, RELEASE=HEAD) 2024-05-03T10:33:32.042 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:31 smithi043 bash[60068]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-c 2024-05-03T10:33:32.043 INFO:journalctl@ceph.mon.c.smithi043.stdout:May 03 10:33:31 smithi043 bash[60094]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon.c" found: no such container 2024-05-03T10:33:32.084 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.c.service' 2024-05-03T10:33:32.125 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:33:32.126 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-05-03T10:33:32.126 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-05-03T10:33:32.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.b 2024-05-03T10:33:32.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:32 smithi181 systemd[1]: Stopping Ceph mon.b for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:33:32.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:32 smithi181 bash[49659]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon.b" found: no such container 2024-05-03T10:33:32.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:32 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-b[23364]: 2024-05-03T10:33:32.331+0000 7f81c56b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:33:32.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:32 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-b[23364]: 2024-05-03T10:33:32.331+0000 7f81c56b2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-03T10:33:32.810 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:32 smithi181 podman[49670]: 2024-05-03 10:33:32.50582945 +0000 UTC m=+0.288282651 container died 5e27fbb569074b97b3d34f6be003c51f311547ad8a1114e57b3182984896ca46 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-b, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, version=8, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.buildah.version=1.19.8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, architecture=x86_64) 2024-05-03T10:33:33.109 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:32 smithi181 podman[49670]: 2024-05-03 10:33:32.898972021 +0000 UTC m=+0.681425220 container cleanup 5e27fbb569074b97b3d34f6be003c51f311547ad8a1114e57b3182984896ca46 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-b, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-type=git, release=754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , architecture=x86_64, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, version=8) 2024-05-03T10:33:33.110 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:32 smithi181 bash[49670]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon-b 2024-05-03T10:33:33.110 INFO:journalctl@ceph.mon.b.smithi181.stdout:May 03 10:33:32 smithi181 bash[49697]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mon.b" found: no such container 2024-05-03T10:33:33.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mon.b.service' 2024-05-03T10:33:33.164 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:33:33.164 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-05-03T10:33:33.165 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-03T10:33:33.165 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mgr.y 2024-05-03T10:33:33.542 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:33 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: ::ffff:172.21.15.181 - - [03/May/2024:10:33:33] "GET /metrics HTTP/1.1" 200 207924 "" "Prometheus/2.33.4" 2024-05-03T10:33:33.543 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:33 smithi043 systemd[1]: Stopping Ceph mgr.y for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:33:33.543 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:33 smithi043 bash[60149]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr.y" found: no such container 2024-05-03T10:33:33.543 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:33 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y[22634]: 2024-05-03T10:33:33.362+0000 7f425f4da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:33:33.543 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:33 smithi043 podman[60162]: 2024-05-03 10:33:33.39468828 +0000 UTC m=+0.149707914 container died e925f3a624c592bc78db21ce091b563e56d1cbeb538847a4d26d9a397693ad7f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, architecture=x86_64, GIT_CLEAN=True, vendor=Red Hat, Inc., io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, release=754, io.k8s.display-name=CentOS Stream 8) 2024-05-03T10:33:34.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:33 smithi043 podman[60162]: 2024-05-03 10:33:33.805363135 +0000 UTC m=+0.560382761 container cleanup e925f3a624c592bc78db21ce091b563e56d1cbeb538847a4d26d9a397693ad7f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, release=754, ceph=True, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8) 2024-05-03T10:33:34.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:33 smithi043 bash[60162]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-y 2024-05-03T10:33:34.042 INFO:journalctl@ceph.mgr.y.smithi043.stdout:May 03 10:33:33 smithi043 bash[60188]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr.y" found: no such container 2024-05-03T10:33:34.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mgr.y.service' 2024-05-03T10:33:34.138 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:33:34.139 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-03T10:33:34.139 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-03T10:33:34.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mgr.x 2024-05-03T10:33:34.619 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:33:34 smithi181 systemd[1]: Stopping Ceph mgr.x for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:33:34.620 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:33:34 smithi181 bash[49750]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr.x" found: no such container 2024-05-03T10:33:34.620 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:33:34 smithi181 podman[49760]: 2024-05-03 10:33:34.386785276 +0000 UTC m=+0.165077358 container died f42a19b78d32214ec5751946c8c0cb07c8c10135179ec9e3cb3cf870ab41a9c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, ceph=True, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.component=centos-stream-container, distribution-scope=public, release=754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-03T10:33:35.031 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:33:34 smithi181 podman[49760]: 2024-05-03 10:33:34.753464467 +0000 UTC m=+0.531756541 container cleanup f42a19b78d32214ec5751946c8c0cb07c8c10135179ec9e3cb3cf870ab41a9c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, version=8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-type=git, name=centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream) 2024-05-03T10:33:35.031 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:33:34 smithi181 bash[49760]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr-x 2024-05-03T10:33:35.031 INFO:journalctl@ceph.mgr.x.smithi181.stdout:May 03 10:33:34 smithi181 bash[49786]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-mgr.x" found: no such container 2024-05-03T10:33:35.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@mgr.x.service' 2024-05-03T10:33:35.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:33:35.073 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-03T10:33:35.074 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-03T10:33:35.074 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.0 2024-05-03T10:33:35.542 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:35 smithi043 systemd[1]: Stopping Ceph osd.0 for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:33:35.542 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:35 smithi043 bash[60243]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.0" found: no such container 2024-05-03T10:33:35.543 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:35 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0[30418]: 2024-05-03T10:33:35.275+0000 7f3981235700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:33:35.543 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:35 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0[30418]: 2024-05-03T10:33:35.275+0000 7f3981235700 -1 osd.0 95 *** Got signal Terminated *** 2024-05-03T10:33:35.543 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:35 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0[30418]: 2024-05-03T10:33:35.275+0000 7f3981235700 -1 osd.0 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T10:33:41.042 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:40 smithi043 podman[60256]: 2024-05-03 10:33:40.753259199 +0000 UTC m=+5.598763295 container died 369dd317255b4d03d31b01740f03e664558ae2fce225f4116aec26683717c928 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.buildah.version=1.19.8, vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, release=754, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=) 2024-05-03T10:33:41.583 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:41 smithi043 podman[60256]: 2024-05-03 10:33:41.397062279 +0000 UTC m=+6.242566368 container cleanup 369dd317255b4d03d31b01740f03e664558ae2fce225f4116aec26683717c928 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, name=centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vcs-type=git, distribution-scope=public, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, maintainer=Guillaume Abrioux , ceph=True) 2024-05-03T10:33:41.583 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:41 smithi043 bash[60256]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0 2024-05-03T10:33:41.993 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:41 smithi043 podman[60303]: 2024-05-03 10:33:41.581449796 +0000 UTC m=+0.829079150 container remove 369dd317255b4d03d31b01740f03e664558ae2fce225f4116aec26683717c928 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0, ceph=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , distribution-scope=public, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.expose-services=, GIT_CLEAN=True, GIT_BRANCH=HEAD, release=754, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8) 2024-05-03T10:33:41.993 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:41 smithi043 bash[60317]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.0" found: no such container 2024-05-03T10:33:42.959 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:42 smithi043 podman[60411]: 2024-05-03 10:33:42.586431547 +0000 UTC m=+0.020737064 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-03T10:33:42.960 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:42 smithi043 podman[60411]: 2024-05-03 10:33:42.723210527 +0000 UTC m=+0.157516034 container create 09125e708f7f575e2f61a8339cf28cf89e5a30d5f24f5c8e1a8a97386181b409 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0-deactivate, release=754, ceph=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, version=8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, io.buildah.version=1.19.8, RELEASE=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-03T10:33:42.960 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:42 smithi043 podman[60411]: 2024-05-03 10:33:42.957844826 +0000 UTC m=+0.392150345 container init 09125e708f7f575e2f61a8339cf28cf89e5a30d5f24f5c8e1a8a97386181b409 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0-deactivate, distribution-scope=public, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.expose-services=, ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, version=8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , architecture=x86_64, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git) 2024-05-03T10:33:43.265 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:42 smithi043 podman[60411]: 2024-05-03 10:33:42.96292962 +0000 UTC m=+0.397235137 container start 09125e708f7f575e2f61a8339cf28cf89e5a30d5f24f5c8e1a8a97386181b409 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0-deactivate, name=centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, release=754, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vcs-type=git, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-05-03T10:33:43.266 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:43 smithi043 podman[60411]: 2024-05-03 10:33:43.057187327 +0000 UTC m=+0.491492849 container attach 09125e708f7f575e2f61a8339cf28cf89e5a30d5f24f5c8e1a8a97386181b409 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0-deactivate, com.redhat.component=centos-stream-container, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, ceph=True, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, name=centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-03T10:33:43.525 INFO:journalctl@ceph.osd.0.smithi043.stdout:May 03 10:33:43 smithi043 podman[60411]: 2024-05-03 10:33:43.343473795 +0000 UTC m=+0.777779311 container died 09125e708f7f575e2f61a8339cf28cf89e5a30d5f24f5c8e1a8a97386181b409 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-0-deactivate, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_BRANCH=HEAD, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container) 2024-05-03T10:33:43.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.0.service' 2024-05-03T10:33:43.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:33:43.963 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-03T10:33:43.963 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-03T10:33:43.963 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.1 2024-05-03T10:33:44.292 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:44 smithi043 systemd[1]: Stopping Ceph osd.1 for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:33:44.792 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:44 smithi043 bash[60547]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.1" found: no such container 2024-05-03T10:33:44.792 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:44 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1[33294]: 2024-05-03T10:33:44.551+0000 7f28f4867700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:33:44.793 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:44 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1[33294]: 2024-05-03T10:33:44.551+0000 7f28f4867700 -1 osd.1 95 *** Got signal Terminated *** 2024-05-03T10:33:44.793 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:44 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1[33294]: 2024-05-03T10:33:44.551+0000 7f28f4867700 -1 osd.1 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T10:33:50.364 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:50 smithi043 podman[60561]: 2024-05-03 10:33:50.082298267 +0000 UTC m=+5.671577537 container died 7924546d3d9f536929943010916881efdb8e745f599b60dca765780a081bd930 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1, architecture=x86_64, ceph=True, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-05-03T10:33:50.757 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:50 smithi043 podman[60561]: 2024-05-03 10:33:50.562695079 +0000 UTC m=+6.151974357 container cleanup 7924546d3d9f536929943010916881efdb8e745f599b60dca765780a081bd930 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, ceph=True, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, release=754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public) 2024-05-03T10:33:50.757 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:50 smithi043 bash[60561]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1 2024-05-03T10:33:51.023 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:50 smithi043 podman[60928]: 2024-05-03 10:33:50.755822635 +0000 UTC m=+0.855656301 container remove 7924546d3d9f536929943010916881efdb8e745f599b60dca765780a081bd930 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, version=8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_BRANCH=HEAD, name=centos-stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, maintainer=Guillaume Abrioux , architecture=x86_64, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, ceph=True) 2024-05-03T10:33:51.023 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:50 smithi043 bash[60978]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.1" found: no such container 2024-05-03T10:33:51.287 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:51 smithi043 podman[61057]: 2024-05-03 10:33:50.927633115 +0000 UTC m=+0.026185430 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-03T10:33:51.542 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:51 smithi043 podman[61057]: 2024-05-03 10:33:51.400874262 +0000 UTC m=+0.499426608 container create daa5f9e4d83b6ba0cae69cd76de592cf6ab2d8951775ccdee72498e46064ec23 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, name=centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, distribution-scope=public, io.buildah.version=1.19.8, RELEASE=HEAD) 2024-05-03T10:33:52.269 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:51 smithi043 podman[61057]: 2024-05-03 10:33:51.963109373 +0000 UTC m=+1.061661674 container init daa5f9e4d83b6ba0cae69cd76de592cf6ab2d8951775ccdee72498e46064ec23 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, version=8, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-type=git, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public) 2024-05-03T10:33:52.270 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:51 smithi043 podman[61057]: 2024-05-03 10:33:51.966931714 +0000 UTC m=+1.065484017 container start daa5f9e4d83b6ba0cae69cd76de592cf6ab2d8951775ccdee72498e46064ec23 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1-deactivate, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_CLEAN=True, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-03T10:33:52.270 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:52 smithi043 podman[61057]: 2024-05-03 10:33:52.046176396 +0000 UTC m=+1.144728712 container attach daa5f9e4d83b6ba0cae69cd76de592cf6ab2d8951775ccdee72498e46064ec23 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , RELEASE=HEAD, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, architecture=x86_64, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, distribution-scope=public, ceph=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-03T10:33:52.542 INFO:journalctl@ceph.osd.1.smithi043.stdout:May 03 10:33:52 smithi043 podman[61057]: 2024-05-03 10:33:52.30195332 +0000 UTC m=+1.400505621 container died daa5f9e4d83b6ba0cae69cd76de592cf6ab2d8951775ccdee72498e46064ec23 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, distribution-scope=public, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., release=754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, vcs-type=git, ceph=True, version=8, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, name=centos-stream, io.openshift.tags=base centos centos-stream) 2024-05-03T10:33:52.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.1.service' 2024-05-03T10:33:53.022 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:33:53.022 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-03T10:33:53.022 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-03T10:33:53.022 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.2 2024-05-03T10:33:53.262 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:53 smithi043 systemd[1]: Stopping Ceph osd.2 for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:33:53.262 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:53 smithi043 bash[61243]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.2" found: no such container 2024-05-03T10:33:54.036 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2[36237]: 2024-05-03T10:33:53.597+0000 7fd46c86b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:33:54.036 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2[36237]: 2024-05-03T10:33:53.597+0000 7fd46c86b700 -1 osd.2 95 *** Got signal Terminated *** 2024-05-03T10:33:54.036 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:53 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2[36237]: 2024-05-03T10:33:53.597+0000 7fd46c86b700 -1 osd.2 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T10:33:59.260 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:58 smithi043 podman[61254]: 2024-05-03 10:33:58.926838114 +0000 UTC m=+5.783598739 container died c7e036ddbc42bbf0955343c6f4243a0a217867381911f222e7b73054820493af (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, ceph=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-03T10:33:59.261 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:59 smithi043 podman[61254]: 2024-05-03 10:33:59.258906136 +0000 UTC m=+6.115666767 container cleanup c7e036ddbc42bbf0955343c6f4243a0a217867381911f222e7b73054820493af (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, version=8, distribution-scope=public, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_BRANCH=HEAD) 2024-05-03T10:33:59.519 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:59 smithi043 bash[61254]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2 2024-05-03T10:33:59.519 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:59 smithi043 podman[61384]: 2024-05-03 10:33:59.46858938 +0000 UTC m=+0.537528843 container remove c7e036ddbc42bbf0955343c6f4243a0a217867381911f222e7b73054820493af (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , version=8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, vcs-type=git, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-05-03T10:33:59.778 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:59 smithi043 bash[61397]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.2" found: no such container 2024-05-03T10:33:59.779 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:59 smithi043 podman[61463]: 2024-05-03 10:33:59.670424195 +0000 UTC m=+0.019458300 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-03T10:34:00.042 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:33:59 smithi043 podman[61463]: 2024-05-03 10:33:59.777643872 +0000 UTC m=+0.126678020 container create 7ade9c6178335de852e6da52f593503ada3ccf328988ec3b91c3affe81e940ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, distribution-scope=public, ceph=True, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, RELEASE=HEAD, version=8, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, maintainer=Guillaume Abrioux , name=centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64) 2024-05-03T10:34:00.042 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:34:00 smithi043 podman[61463]: 2024-05-03 10:34:00.01344274 +0000 UTC m=+0.362476846 container init 7ade9c6178335de852e6da52f593503ada3ccf328988ec3b91c3affe81e940ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2-deactivate, architecture=x86_64, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.openshift.expose-services=, RELEASE=HEAD, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-03T10:34:00.042 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:34:00 smithi043 podman[61463]: 2024-05-03 10:34:00.016696379 +0000 UTC m=+0.365730487 container start 7ade9c6178335de852e6da52f593503ada3ccf328988ec3b91c3affe81e940ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2-deactivate, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.expose-services=, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, version=8, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-03T10:34:00.542 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:34:00 smithi043 podman[61463]: 2024-05-03 10:34:00.088387807 +0000 UTC m=+0.437421920 container attach 7ade9c6178335de852e6da52f593503ada3ccf328988ec3b91c3affe81e940ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=-17.2.0, ceph=True, version=8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-type=git, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8) 2024-05-03T10:34:00.542 INFO:journalctl@ceph.osd.2.smithi043.stdout:May 03 10:34:00 smithi043 podman[61463]: 2024-05-03 10:34:00.306689231 +0000 UTC m=+0.655723336 container died 7ade9c6178335de852e6da52f593503ada3ccf328988ec3b91c3affe81e940ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-2-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, distribution-scope=public, name=centos-stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870) 2024-05-03T10:34:00.872 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.2.service' 2024-05-03T10:34:00.914 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:34:00.914 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-03T10:34:00.914 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-03T10:34:00.914 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.3 2024-05-03T10:34:01.113 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:00 smithi043 systemd[1]: Stopping Ceph osd.3 for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:34:01.113 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:01 smithi043 bash[61542]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.3" found: no such container 2024-05-03T10:34:01.521 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:01 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3[39290]: 2024-05-03T10:34:01.110+0000 7fb3c79fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:34:01.522 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:01 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3[39290]: 2024-05-03T10:34:01.110+0000 7fb3c79fa700 -1 osd.3 95 *** Got signal Terminated *** 2024-05-03T10:34:01.522 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:01 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3[39290]: 2024-05-03T10:34:01.110+0000 7fb3c79fa700 -1 osd.3 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T10:34:03.120 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:02 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:02.849+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:03.619 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:03 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:03.358+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:04.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:03 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:03.845+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:04.869 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:04 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:04.377+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:05.369 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:04 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:04.883+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:39.280512+0000 front 2024-05-03T10:33:39.280423+0000 (oldest deadline 2024-05-03T10:34:04.580282+0000) 2024-05-03T10:34:05.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:04 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:04.878+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:05.869 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:05 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:05.381+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:05.869 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:05 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:05.552+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:06.370 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:05 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:05.933+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:39.280512+0000 front 2024-05-03T10:33:39.280423+0000 (oldest deadline 2024-05-03T10:34:04.580282+0000) 2024-05-03T10:34:06.370 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:05 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:05.873+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:06.784 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:06 smithi043 podman[61553]: 2024-05-03 10:34:06.508324565 +0000 UTC m=+5.482625407 container died 5632c3f6a0a3f27e6e5751b1b011292d63624fb91356fd3f7ec342d3c829102a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , RELEASE=HEAD, distribution-scope=public, io.openshift.expose-services=, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, release=754, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0) 2024-05-03T10:34:06.869 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:06 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:06.395+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:06.869 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:06 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:06.524+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:07.042 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:06 smithi043 podman[61553]: 2024-05-03 10:34:06.872865386 +0000 UTC m=+5.847166228 container cleanup 5632c3f6a0a3f27e6e5751b1b011292d63624fb91356fd3f7ec342d3c829102a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, ceph=True, vcs-type=git, name=centos-stream, release=754, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.openshift.expose-services=, vendor=Red Hat, Inc., io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-05-03T10:34:07.042 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:06 smithi043 bash[61553]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3 2024-05-03T10:34:07.042 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:06 smithi043 bash[61694]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.3" found: no such container 2024-05-03T10:34:07.266 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:06 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:06.939+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:39.280512+0000 front 2024-05-03T10:33:39.280423+0000 (oldest deadline 2024-05-03T10:34:04.580282+0000) 2024-05-03T10:34:07.266 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:06 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:06.884+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:07.351 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:07 smithi043 podman[61681]: 2024-05-03 10:34:07.090855607 +0000 UTC m=+0.584653804 container remove 5632c3f6a0a3f27e6e5751b1b011292d63624fb91356fd3f7ec342d3c829102a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3, CEPH_POINT_RELEASE=-17.2.0, version=8, io.openshift.expose-services=, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, RELEASE=HEAD, com.redhat.component=centos-stream-container, release=754, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870) 2024-05-03T10:34:07.352 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:07 smithi043 podman[61760]: 2024-05-03 10:34:07.258251924 +0000 UTC m=+0.105757975 container create d34169f87dbf313fbe02bca19f3d4aa5722ce853605bff293bd145beb8410a62 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3-deactivate, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, RELEASE=HEAD, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-05-03T10:34:07.352 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:07 smithi043 podman[61760]: 2024-05-03 10:34:07.172702852 +0000 UTC m=+0.020208867 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-03T10:34:07.554 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:07 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:07.408+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:07.763 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:07 smithi043 podman[61760]: 2024-05-03 10:34:07.469115847 +0000 UTC m=+0.316621860 container init d34169f87dbf313fbe02bca19f3d4aa5722ce853605bff293bd145beb8410a62 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, name=centos-stream, io.k8s.display-name=CentOS Stream 8, release=754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, io.buildah.version=1.19.8) 2024-05-03T10:34:07.763 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:07 smithi043 podman[61760]: 2024-05-03 10:34:07.472601215 +0000 UTC m=+0.320107232 container start d34169f87dbf313fbe02bca19f3d4aa5722ce853605bff293bd145beb8410a62 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3-deactivate, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, name=centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.buildah.version=1.19.8, distribution-scope=public, version=8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_BRANCH=HEAD) 2024-05-03T10:34:07.763 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:07 smithi043 podman[61760]: 2024-05-03 10:34:07.535235737 +0000 UTC m=+0.382741769 container attach d34169f87dbf313fbe02bca19f3d4aa5722ce853605bff293bd145beb8410a62 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vcs-type=git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8, distribution-scope=public, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64) 2024-05-03T10:34:07.850 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:07 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:07.551+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:08.042 INFO:journalctl@ceph.osd.3.smithi043.stdout:May 03 10:34:07 smithi043 podman[61760]: 2024-05-03 10:34:07.762609792 +0000 UTC m=+0.610115814 container died d34169f87dbf313fbe02bca19f3d4aa5722ce853605bff293bd145beb8410a62 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-3-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, release=754, architecture=x86_64, io.openshift.expose-services=, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-05-03T10:34:08.119 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:07 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:07.978+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:39.280512+0000 front 2024-05-03T10:33:39.280423+0000 (oldest deadline 2024-05-03T10:34:04.580282+0000) 2024-05-03T10:34:08.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:07 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:07.872+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:08.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.3.service' 2024-05-03T10:34:08.429 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:34:08.429 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-03T10:34:08.429 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-03T10:34:08.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.4 2024-05-03T10:34:08.863 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:08 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:08.376+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:08.864 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:08 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:08.552+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:08.864 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:08 smithi181 systemd[1]: Stopping Ceph osd.4 for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:34:08.864 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:08 smithi181 bash[50608]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.4" found: no such container 2024-05-03T10:34:08.864 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:08 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:08.584+0000 7fd22cc4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:34:08.864 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:08 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:08.584+0000 7fd22cc4b700 -1 osd.4 95 *** Got signal Terminated *** 2024-05-03T10:34:08.864 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:08 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:08.584+0000 7fd22cc4b700 -1 osd.4 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T10:34:09.119 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:08 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:08.965+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:39.280512+0000 front 2024-05-03T10:33:39.280423+0000 (oldest deadline 2024-05-03T10:34:04.580282+0000) 2024-05-03T10:34:09.120 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:08 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:08.861+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:09.705 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:09.383+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:09.705 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:09.504+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:10.119 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:09.933+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:39.280512+0000 front 2024-05-03T10:33:39.280423+0000 (oldest deadline 2024-05-03T10:34:04.580282+0000) 2024-05-03T10:34:10.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:09 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:09.862+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:10.869 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:10 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:10.428+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:10.869 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:10 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:10.528+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:11.369 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:10 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:10.946+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:39.280512+0000 front 2024-05-03T10:33:39.280423+0000 (oldest deadline 2024-05-03T10:34:04.580282+0000) 2024-05-03T10:34:11.369 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:10 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:10.946+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:48.581695+0000 front 2024-05-03T10:33:48.581679+0000 (oldest deadline 2024-05-03T10:34:10.881710+0000) 2024-05-03T10:34:11.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:10 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:10.884+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:11.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:10 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:10.884+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:11.842 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:11 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:11.451+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:11.843 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:11 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:11.523+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:12.119 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:11 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:11.989+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:39.280512+0000 front 2024-05-03T10:33:39.280423+0000 (oldest deadline 2024-05-03T10:34:04.580282+0000) 2024-05-03T10:34:12.119 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:11 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:11.989+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:48.581695+0000 front 2024-05-03T10:33:48.581679+0000 (oldest deadline 2024-05-03T10:34:10.881710+0000) 2024-05-03T10:34:12.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:11 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:11.840+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:12.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:11 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:11.840+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:12.869 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:12 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:12.407+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:12.870 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:12 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:12.502+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:13.364 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:12 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:12.876+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:13.364 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:12 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:12.876+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:13.365 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:12 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:12.943+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:39.280512+0000 front 2024-05-03T10:33:39.280423+0000 (oldest deadline 2024-05-03T10:34:04.580282+0000) 2024-05-03T10:34:13.365 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:12 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4[28085]: 2024-05-03T10:34:12.943+0000 7fd229de7700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:48.581695+0000 front 2024-05-03T10:33:48.581679+0000 (oldest deadline 2024-05-03T10:34:10.881710+0000) 2024-05-03T10:34:13.619 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:13 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:13.362+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:13.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:13 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:13.526+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:14.121 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:13 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:13.861+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:14.122 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:13 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:13.861+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:14.122 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:14 smithi181 podman[50620]: 2024-05-03 10:34:14.112532407 +0000 UTC m=+5.604830074 container died 383cc066a124d138611a69fb762fbec91bf25954c162fd3401a1d1c9fe54811f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=HEAD, io.openshift.expose-services=, version=8, GIT_CLEAN=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.tags=base centos centos-stream) 2024-05-03T10:34:14.619 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:14 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:14.353+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:14.620 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:14 smithi181 podman[50620]: 2024-05-03 10:34:14.468678384 +0000 UTC m=+5.960976042 container cleanup 383cc066a124d138611a69fb762fbec91bf25954c162fd3401a1d1c9fe54811f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, distribution-scope=public, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-03T10:34:14.620 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:14 smithi181 bash[50620]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4 2024-05-03T10:34:14.620 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:14 smithi181 bash[50661]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.4" found: no such container 2024-05-03T10:34:14.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:14 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:14.476+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:14.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:14 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:14.476+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:15.053 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:14 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:14.821+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:15.053 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:14 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:14.821+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:15.054 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:14 smithi181 podman[50649]: 2024-05-03 10:34:14.74529548 +0000 UTC m=+0.630194144 container remove 383cc066a124d138611a69fb762fbec91bf25954c162fd3401a1d1c9fe54811f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, GIT_CLEAN=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, release=754, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870) 2024-05-03T10:34:15.054 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:14 smithi181 podman[50724]: 2024-05-03 10:34:14.81876158 +0000 UTC m=+0.019562700 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-03T10:34:15.054 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:14 smithi181 podman[50724]: 2024-05-03 10:34:14.954561655 +0000 UTC m=+0.155362805 container create 95becee3b69d781c827e1ffc9c05590c9e8cf815c1b42d2c8269ffcf7935a48c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4-deactivate, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, version=8, architecture=x86_64, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8) 2024-05-03T10:34:15.350 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:15 smithi181 podman[50724]: 2024-05-03 10:34:15.207062429 +0000 UTC m=+0.407863536 container init 95becee3b69d781c827e1ffc9c05590c9e8cf815c1b42d2c8269ffcf7935a48c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4-deactivate, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, RELEASE=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64) 2024-05-03T10:34:15.350 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:15 smithi181 podman[50724]: 2024-05-03 10:34:15.210402976 +0000 UTC m=+0.411204081 container start 95becee3b69d781c827e1ffc9c05590c9e8cf815c1b42d2c8269ffcf7935a48c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4-deactivate, architecture=x86_64, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., release=754, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, distribution-scope=public, ceph=True, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, RELEASE=HEAD, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container) 2024-05-03T10:34:15.351 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:15 smithi181 podman[50724]: 2024-05-03 10:34:15.281964245 +0000 UTC m=+0.482765350 container attach 95becee3b69d781c827e1ffc9c05590c9e8cf815c1b42d2c8269ffcf7935a48c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.buildah.version=1.19.8, release=754, GIT_CLEAN=True, name=centos-stream, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-03T10:34:15.619 INFO:journalctl@ceph.osd.4.smithi181.stdout:May 03 10:34:15 smithi181 podman[50724]: 2024-05-03 10:34:15.495506299 +0000 UTC m=+0.696307420 container died 95becee3b69d781c827e1ffc9c05590c9e8cf815c1b42d2c8269ffcf7935a48c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-4-deactivate, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_CLEAN=True, version=8, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, name=centos-stream, vcs-type=git, RELEASE=HEAD, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, ceph=True) 2024-05-03T10:34:15.619 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:15 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:15.348+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:15.620 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:15 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:15.348+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.543507+0000 front 2024-05-03T10:33:44.543501+0000 (oldest deadline 2024-05-03T10:34:14.543823+0000) 2024-05-03T10:34:15.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:15 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:15.458+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:15.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:15 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:15.458+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:16.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:15 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:15.813+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:16.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:15 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:15.813+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:16.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.4.service' 2024-05-03T10:34:16.295 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:34:16.295 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-03T10:34:16.295 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-03T10:34:16.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.5 2024-05-03T10:34:16.523 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:16 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:16.433+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:16.523 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:16 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:16.433+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:16.524 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:16 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:16.318+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:16.524 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:16 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:16.318+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.543507+0000 front 2024-05-03T10:33:44.543501+0000 (oldest deadline 2024-05-03T10:34:14.543823+0000) 2024-05-03T10:34:16.524 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:16 smithi181 systemd[1]: Stopping Ceph osd.5 for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:34:16.524 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:16 smithi181 bash[50803]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.5" found: no such container 2024-05-03T10:34:16.790 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:16 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:16.521+0000 7f743de9c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:34:16.791 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:16 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:16.521+0000 7f743de9c700 -1 osd.5 95 *** Got signal Terminated *** 2024-05-03T10:34:16.791 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:16 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:16.521+0000 7f743de9c700 -1 osd.5 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T10:34:17.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:16 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:16.789+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:17.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:16 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:16.789+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:17.622 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:17 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:17.315+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:17.622 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:17 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:17.315+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.543507+0000 front 2024-05-03T10:33:44.543501+0000 (oldest deadline 2024-05-03T10:34:14.543823+0000) 2024-05-03T10:34:17.622 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:17 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:17.414+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:17.623 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:17 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:17.414+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:18.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:17 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:17.793+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:18.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:17 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:17.793+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:18.619 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:18 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:18.296+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:18.619 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:18 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:18.296+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.543507+0000 front 2024-05-03T10:33:44.543501+0000 (oldest deadline 2024-05-03T10:34:14.543823+0000) 2024-05-03T10:34:18.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:18 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:18.412+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:18.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:18 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:18.412+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:19.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:18 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:18.786+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:19.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:18 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:18.786+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:19.619 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:19.307+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:19.619 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:19.307+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.543507+0000 front 2024-05-03T10:33:44.543501+0000 (oldest deadline 2024-05-03T10:34:14.543823+0000) 2024-05-03T10:34:19.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:19.457+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:19.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:19.457+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:20.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:19.801+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:20.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:19 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:19.801+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:20.619 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:20.335+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:20.620 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:20.335+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.543507+0000 front 2024-05-03T10:33:44.543501+0000 (oldest deadline 2024-05-03T10:34:14.543823+0000) 2024-05-03T10:34:20.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:20.451+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:20.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:20.451+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:21.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:20.774+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:21.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:20.774+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:21.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:20 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:20.774+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:21.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:21.473+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:21.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:21.473+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:21.620 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:21.291+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:38.341715+0000 front 2024-05-03T10:33:38.341601+0000 (oldest deadline 2024-05-03T10:34:02.341979+0000) 2024-05-03T10:34:21.620 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:21.291+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.543507+0000 front 2024-05-03T10:33:44.543501+0000 (oldest deadline 2024-05-03T10:34:14.543823+0000) 2024-05-03T10:34:21.620 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5[30985]: 2024-05-03T10:34:21.291+0000 7f7439034700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:54.544335+0000 front 2024-05-03T10:33:54.544383+0000 (oldest deadline 2024-05-03T10:34:20.944649+0000) 2024-05-03T10:34:22.119 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:22 smithi181 podman[50816]: 2024-05-03 10:34:22.049212352 +0000 UTC m=+5.644927864 container died bf96e37f9fadf22a561e0fe3319275055463508896110262a82c063d9fd2fe5e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.k8s.display-name=CentOS Stream 8, name=centos-stream) 2024-05-03T10:34:22.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:21.803+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:22.120 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:21.803+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:22.120 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:21 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:21.803+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:22.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:22 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:22.469+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:22.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:22 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:22.469+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:22.620 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:22 smithi181 podman[50816]: 2024-05-03 10:34:22.505046852 +0000 UTC m=+6.100762362 container cleanup bf96e37f9fadf22a561e0fe3319275055463508896110262a82c063d9fd2fe5e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, version=8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, release=754, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8) 2024-05-03T10:34:22.620 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:22 smithi181 bash[50816]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5 2024-05-03T10:34:22.620 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:22 smithi181 bash[50846]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.5" found: no such container 2024-05-03T10:34:23.024 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:22 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:22.821+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:23.024 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:22 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:22.821+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:23.024 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:22 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:22.821+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:23.024 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:22 smithi181 podman[50833]: 2024-05-03 10:34:22.756440373 +0000 UTC m=+0.698430424 container remove bf96e37f9fadf22a561e0fe3319275055463508896110262a82c063d9fd2fe5e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , version=8, io.openshift.tags=base centos centos-stream, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, RELEASE=HEAD, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64) 2024-05-03T10:34:23.025 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:22 smithi181 podman[50913]: 2024-05-03 10:34:22.808298609 +0000 UTC m=+0.018867853 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-03T10:34:23.025 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:22 smithi181 podman[50913]: 2024-05-03 10:34:22.924052046 +0000 UTC m=+0.134621301 container create 72efbf16fb0c208909d215a7f2cf67037a3f32107083d25d546b4e8a8aec9c5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5-deactivate, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , ceph=True, distribution-scope=public, architecture=x86_64, release=754, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.buildah.version=1.19.8) 2024-05-03T10:34:23.370 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:23 smithi181 podman[50913]: 2024-05-03 10:34:23.176351897 +0000 UTC m=+0.386921114 container init 72efbf16fb0c208909d215a7f2cf67037a3f32107083d25d546b4e8a8aec9c5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5-deactivate, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, ceph=True, version=8, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True) 2024-05-03T10:34:23.370 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:23 smithi181 podman[50913]: 2024-05-03 10:34:23.179677278 +0000 UTC m=+0.390246494 container start 72efbf16fb0c208909d215a7f2cf67037a3f32107083d25d546b4e8a8aec9c5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5-deactivate, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.buildah.version=1.19.8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, version=8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-03T10:34:23.370 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:23 smithi181 podman[50913]: 2024-05-03 10:34:23.259581851 +0000 UTC m=+0.470151084 container attach 72efbf16fb0c208909d215a7f2cf67037a3f32107083d25d546b4e8a8aec9c5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5-deactivate, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, version=8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, vcs-type=git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, distribution-scope=public) 2024-05-03T10:34:23.805 INFO:journalctl@ceph.osd.5.smithi181.stdout:May 03 10:34:23 smithi181 podman[50913]: 2024-05-03 10:34:23.459174271 +0000 UTC m=+0.669743488 container died 72efbf16fb0c208909d215a7f2cf67037a3f32107083d25d546b4e8a8aec9c5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, name=centos-stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, version=8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , distribution-scope=public, vendor=Red Hat, Inc., release=754, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0) 2024-05-03T10:34:23.806 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:23 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:23.517+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:23.806 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:23 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:23.517+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:24.114 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:23 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:23.815+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:24.114 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:23 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:23.815+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:24.114 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:23 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:23.815+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:24.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.5.service' 2024-05-03T10:34:24.177 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:34:24.177 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-03T10:34:24.178 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-03T10:34:24.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.6 2024-05-03T10:34:24.369 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:24 smithi181 systemd[1]: Stopping Ceph osd.6 for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:34:24.369 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:24 smithi181 bash[50991]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.6" found: no such container 2024-05-03T10:34:24.867 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:24.407+0000 7f47eae82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:34:24.867 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:24.407+0000 7f47eae82700 -1 osd.6 95 *** Got signal Terminated *** 2024-05-03T10:34:24.867 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:24.407+0000 7f47eae82700 -1 osd.6 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T10:34:24.868 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:24.480+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:24.868 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:24.480+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:25.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:24.865+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:25.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:24.865+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:25.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:24 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:24.865+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:25.869 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:25.453+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:25.869 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:25.453+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:25.869 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:25.453+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:53.555040+0000 front 2024-05-03T10:33:53.555077+0000 (oldest deadline 2024-05-03T10:34:24.755501+0000) 2024-05-03T10:34:26.304 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:25.900+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:26.304 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:25.900+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:26.305 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:25 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:25.900+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:26.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:26 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:26.416+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:26.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:26 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:26.416+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:26.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:26 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:26.416+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:53.555040+0000 front 2024-05-03T10:33:53.555077+0000 (oldest deadline 2024-05-03T10:34:24.755501+0000) 2024-05-03T10:34:27.118 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:26 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:26.858+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:27.118 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:26 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:26.858+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:27.118 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:26 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:26.858+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:27.869 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:27 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:27.410+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:27.869 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:27 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:27.410+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:27.870 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:27 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:27.410+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:53.555040+0000 front 2024-05-03T10:33:53.555077+0000 (oldest deadline 2024-05-03T10:34:24.755501+0000) 2024-05-03T10:34:28.148 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:27 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:27.889+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:28.149 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:27 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:27.889+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:28.149 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:27 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:27.889+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:28.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:28 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:28.365+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:28.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:28 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:28.365+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:28.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:28 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:28.365+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:53.555040+0000 front 2024-05-03T10:33:53.555077+0000 (oldest deadline 2024-05-03T10:34:24.755501+0000) 2024-05-03T10:34:28.620 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:28 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:28.365+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.756077+0000 front 2024-05-03T10:34:04.756004+0000 (oldest deadline 2024-05-03T10:34:27.655912+0000) 2024-05-03T10:34:29.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:28 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:28.849+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:29.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:28 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:28.849+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:29.119 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:28 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:28.849+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:29.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:29.321+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:40.053730+0000 front 2024-05-03T10:33:40.053660+0000 (oldest deadline 2024-05-03T10:34:05.353627+0000) 2024-05-03T10:34:29.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:29.321+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:49.454500+0000 front 2024-05-03T10:33:49.454466+0000 (oldest deadline 2024-05-03T10:34:13.554547+0000) 2024-05-03T10:34:29.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:29.321+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:53.555040+0000 front 2024-05-03T10:33:53.555077+0000 (oldest deadline 2024-05-03T10:34:24.755501+0000) 2024-05-03T10:34:29.619 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6[33891]: 2024-05-03T10:34:29.321+0000 7f47e601a700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.756077+0000 front 2024-05-03T10:34:04.756004+0000 (oldest deadline 2024-05-03T10:34:27.655912+0000) 2024-05-03T10:34:30.046 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:29 smithi181 podman[51002]: 2024-05-03 10:34:29.774801047 +0000 UTC m=+5.484565162 container died 8d9702f93d88637956dfd15907ca420315d54b1925c43a82608e530179129bdf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, release=754, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-03T10:34:30.047 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:29.899+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:30.047 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:29.899+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:30.047 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:29.899+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:30.047 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:29 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:29.899+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.807207+0000 front 2024-05-03T10:34:04.807231+0000 (oldest deadline 2024-05-03T10:34:28.907216+0000) 2024-05-03T10:34:30.369 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:30 smithi181 podman[51002]: 2024-05-03 10:34:30.143906988 +0000 UTC m=+5.853671112 container cleanup 8d9702f93d88637956dfd15907ca420315d54b1925c43a82608e530179129bdf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6, version=8, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, ceph=True, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., RELEASE=HEAD, release=754, name=centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux ) 2024-05-03T10:34:30.369 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:30 smithi181 bash[51002]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6 2024-05-03T10:34:30.369 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:30 smithi181 bash[51521]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.6" found: no such container 2024-05-03T10:34:30.869 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:30 smithi181 podman[51480]: 2024-05-03 10:34:30.411823469 +0000 UTC m=+0.656049568 container remove 8d9702f93d88637956dfd15907ca420315d54b1925c43a82608e530179129bdf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, name=centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_BRANCH=HEAD, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, release=754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc.) 2024-05-03T10:34:31.201 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:30 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:30.894+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:31.201 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:30 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:30.894+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:31.201 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:30 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:30.894+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:31.202 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:30 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:30.894+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.807207+0000 front 2024-05-03T10:34:04.807231+0000 (oldest deadline 2024-05-03T10:34:28.907216+0000) 2024-05-03T10:34:31.202 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:30 smithi181 podman[51619]: 2024-05-03 10:34:30.83600545 +0000 UTC m=+0.019874871 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-03T10:34:31.202 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:30 smithi181 podman[51619]: 2024-05-03 10:34:30.956249887 +0000 UTC m=+0.140119335 container create 89ffec648aa3dd589485a177ea44e630cdb3e39ec0c80bed5ef6afa14c44a959 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , release=754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, distribution-scope=public, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.component=centos-stream-container) 2024-05-03T10:34:31.202 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:31 smithi181 podman[51619]: 2024-05-03 10:34:31.200115011 +0000 UTC m=+0.383984416 container init 89ffec648aa3dd589485a177ea44e630cdb3e39ec0c80bed5ef6afa14c44a959 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6-deactivate, vcs-type=git, name=centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, version=8, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-05-03T10:34:31.486 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:31 smithi181 podman[51619]: 2024-05-03 10:34:31.203565578 +0000 UTC m=+0.387434987 container start 89ffec648aa3dd589485a177ea44e630cdb3e39ec0c80bed5ef6afa14c44a959 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6-deactivate, name=centos-stream, vcs-type=git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vendor=Red Hat, Inc., io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, version=8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-03T10:34:31.487 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:31 smithi181 podman[51619]: 2024-05-03 10:34:31.308443573 +0000 UTC m=+0.492312979 container attach 89ffec648aa3dd589485a177ea44e630cdb3e39ec0c80bed5ef6afa14c44a959 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, GIT_CLEAN=True, architecture=x86_64, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vcs-type=git, maintainer=Guillaume Abrioux ) 2024-05-03T10:34:32.119 INFO:journalctl@ceph.osd.6.smithi181.stdout:May 03 10:34:32 smithi181 podman[51619]: 2024-05-03 10:34:32.015444134 +0000 UTC m=+1.199313534 container died 89ffec648aa3dd589485a177ea44e630cdb3e39ec0c80bed5ef6afa14c44a959 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-6-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.openshift.expose-services=, maintainer=Guillaume Abrioux , name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_CLEAN=True, vendor=Red Hat, Inc., ceph=True, io.buildah.version=1.19.8, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream) 2024-05-03T10:34:32.120 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:31 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:31.930+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:32.120 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:31 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:31.930+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:32.120 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:31 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:31.930+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:32.120 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:31 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:31.930+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.807207+0000 front 2024-05-03T10:34:04.807231+0000 (oldest deadline 2024-05-03T10:34:28.907216+0000) 2024-05-03T10:34:32.633 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.6.service' 2024-05-03T10:34:32.679 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:34:32.680 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-03T10:34:32.680 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-03T10:34:32.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.7 2024-05-03T10:34:32.869 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:32 smithi181 systemd[1]: Stopping Ceph osd.7 for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:34:33.355 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:32 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:32.947+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:33.356 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:32 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:32.947+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:33.356 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:32 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:32.947+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:33.356 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:32 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:32.947+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.807207+0000 front 2024-05-03T10:34:04.807231+0000 (oldest deadline 2024-05-03T10:34:28.907216+0000) 2024-05-03T10:34:33.356 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:32 smithi181 bash[51784]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.7" found: no such container 2024-05-03T10:34:33.356 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:33 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:33.353+0000 7f85bfd86700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T10:34:33.356 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:33 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:33.353+0000 7f85bfd86700 -1 osd.7 95 *** Got signal Terminated *** 2024-05-03T10:34:33.356 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:33 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:33.353+0000 7f85bfd86700 -1 osd.7 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T10:34:34.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:33 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:33.920+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:34.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:33 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:33.920+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:34.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:33 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:33.920+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:34.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:33 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:33.920+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.807207+0000 front 2024-05-03T10:34:04.807231+0000 (oldest deadline 2024-05-03T10:34:28.907216+0000) 2024-05-03T10:34:35.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:34 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:34.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:35.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:34 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:34.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:35.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:34 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:34.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:35.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:34 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:34.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.807207+0000 front 2024-05-03T10:34:04.807231+0000 (oldest deadline 2024-05-03T10:34:28.907216+0000) 2024-05-03T10:34:35.370 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:34 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:34.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.181:6804 osd.4 since back 2024-05-03T10:34:09.408183+0000 front 2024-05-03T10:34:09.408072+0000 (oldest deadline 2024-05-03T10:34:34.708149+0000) 2024-05-03T10:34:36.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:35 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:35.876+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:36.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:35 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:35.876+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:36.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:35 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:35.876+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:36.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:35 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:35.876+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.807207+0000 front 2024-05-03T10:34:04.807231+0000 (oldest deadline 2024-05-03T10:34:28.907216+0000) 2024-05-03T10:34:36.370 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:35 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:35.876+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.181:6804 osd.4 since back 2024-05-03T10:34:09.408183+0000 front 2024-05-03T10:34:09.408072+0000 (oldest deadline 2024-05-03T10:34:34.708149+0000) 2024-05-03T10:34:36.370 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:35 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:35.876+0000 7f85bcf22700 -1 osd.7 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24604.0:92 2.0 2:029cf94b:::realms.b22db950-3b56-45cd-a14b-b934aaa70506.control:head [watch ping cookie 94708770778496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) 2024-05-03T10:34:37.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:36 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:36.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:37.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:36 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:36.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:37.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:36 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:36.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:37.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:36 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:36.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.807207+0000 front 2024-05-03T10:34:04.807231+0000 (oldest deadline 2024-05-03T10:34:28.907216+0000) 2024-05-03T10:34:37.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:36 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:36.912+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.181:6804 osd.4 since back 2024-05-03T10:34:09.408183+0000 front 2024-05-03T10:34:09.408072+0000 (oldest deadline 2024-05-03T10:34:34.708149+0000) 2024-05-03T10:34:37.370 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:36 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:36.912+0000 7f85bcf22700 -1 osd.7 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24604.0:92 2.0 2:029cf94b:::realms.b22db950-3b56-45cd-a14b-b934aaa70506.control:head [watch ping cookie 94708770778496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) 2024-05-03T10:34:38.270 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:37 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:37.943+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6806 osd.0 since back 2024-05-03T10:33:36.704192+0000 front 2024-05-03T10:33:36.704264+0000 (oldest deadline 2024-05-03T10:34:02.604066+0000) 2024-05-03T10:34:38.271 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:37 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:37.943+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6814 osd.1 since back 2024-05-03T10:33:44.304941+0000 front 2024-05-03T10:33:44.305050+0000 (oldest deadline 2024-05-03T10:34:10.205012+0000) 2024-05-03T10:34:38.271 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:37 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:37.943+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6822 osd.2 since back 2024-05-03T10:33:56.606476+0000 front 2024-05-03T10:33:56.606504+0000 (oldest deadline 2024-05-03T10:34:20.706365+0000) 2024-05-03T10:34:38.271 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:37 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:37.943+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.43:6830 osd.3 since back 2024-05-03T10:34:04.807207+0000 front 2024-05-03T10:34:04.807231+0000 (oldest deadline 2024-05-03T10:34:28.907216+0000) 2024-05-03T10:34:38.272 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:37 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:37.943+0000 7f85bcf22700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.181:6804 osd.4 since back 2024-05-03T10:34:09.408183+0000 front 2024-05-03T10:34:09.408072+0000 (oldest deadline 2024-05-03T10:34:34.708149+0000) 2024-05-03T10:34:38.272 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:37 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7[36957]: 2024-05-03T10:34:37.943+0000 7f85bcf22700 -1 osd.7 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24604.0:92 2.0 2:029cf94b:::realms.b22db950-3b56-45cd-a14b-b934aaa70506.control:head [watch ping cookie 94708770778496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) 2024-05-03T10:34:39.156 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:38 smithi181 podman[51824]: 2024-05-03 10:34:38.874001021 +0000 UTC m=+5.890504319 container died 151fde494c401cc30b3a41d2a1e79b5d89c4f4fd7cc03d259fc5b3af61274409 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, ceph=True, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-03T10:34:39.494 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:39 smithi181 podman[51824]: 2024-05-03 10:34:39.303403955 +0000 UTC m=+6.319907252 container cleanup 151fde494c401cc30b3a41d2a1e79b5d89c4f4fd7cc03d259fc5b3af61274409 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, architecture=x86_64, io.openshift.expose-services=, name=centos-stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , distribution-scope=public, version=8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True) 2024-05-03T10:34:39.494 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:39 smithi181 bash[51824]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7 2024-05-03T10:34:39.495 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:39 smithi181 bash[51899]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd.7" found: no such container 2024-05-03T10:34:39.765 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:39 smithi181 podman[51884]: 2024-05-03 10:34:39.538292371 +0000 UTC m=+0.661138441 container remove 151fde494c401cc30b3a41d2a1e79b5d89c4f4fd7cc03d259fc5b3af61274409 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7, vcs-type=git, name=centos-stream, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, architecture=x86_64, io.buildah.version=1.19.8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-05-03T10:34:39.765 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:39 smithi181 podman[51973]: 2024-05-03 10:34:39.638717641 +0000 UTC m=+0.019017948 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-03T10:34:39.765 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:39 smithi181 podman[51973]: 2024-05-03 10:34:39.764169731 +0000 UTC m=+0.144470038 container create 4035fa742b077a9de992af15921a7f2ec29fa2b95ea6a0827d2c8ce343a95517 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7-deactivate, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, RELEASE=HEAD, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, ceph=True, GIT_CLEAN=True, release=754, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.component=centos-stream-container) 2024-05-03T10:34:40.051 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:39 smithi181 podman[51973]: 2024-05-03 10:34:39.975158203 +0000 UTC m=+0.355458520 container init 4035fa742b077a9de992af15921a7f2ec29fa2b95ea6a0827d2c8ce343a95517 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7-deactivate, name=centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, architecture=x86_64, RELEASE=HEAD, io.openshift.expose-services=, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-03T10:34:40.051 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:39 smithi181 podman[51973]: 2024-05-03 10:34:39.978634706 +0000 UTC m=+0.358935012 container start 4035fa742b077a9de992af15921a7f2ec29fa2b95ea6a0827d2c8ce343a95517 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, RELEASE=HEAD, vendor=Red Hat, Inc., release=754, ceph=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-05-03T10:34:40.051 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:40 smithi181 podman[51973]: 2024-05-03 10:34:40.049720602 +0000 UTC m=+0.430020940 container attach 4035fa742b077a9de992af15921a7f2ec29fa2b95ea6a0827d2c8ce343a95517 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, io.buildah.version=1.19.8, RELEASE=HEAD, name=centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, maintainer=Guillaume Abrioux , version=8) 2024-05-03T10:34:40.369 INFO:journalctl@ceph.osd.7.smithi181.stdout:May 03 10:34:40 smithi181 podman[51973]: 2024-05-03 10:34:40.276612681 +0000 UTC m=+0.656912981 container died 4035fa742b077a9de992af15921a7f2ec29fa2b95ea6a0827d2c8ce343a95517 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-osd-7-deactivate, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, release=754, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, ceph=True, RELEASE=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_CLEAN=True, name=centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-03T10:34:40.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@osd.7.service' 2024-05-03T10:34:40.895 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:34:40.895 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-03T10:34:40.895 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-05-03T10:34:40.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-79bfd44e-0936-11ef-bc97-c7b262605968@prometheus.a 2024-05-03T10:34:41.099 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:40 smithi181 systemd[1]: Stopping Ceph prometheus.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:34:41.099 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:40 smithi181 bash[52055]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-03T10:34:41.369 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.097Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-03T10:34:41.369 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.097Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-03T10:34:41.369 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.097Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-03T10:34:41.370 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.097Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-03T10:34:41.370 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.097Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-03T10:34:41.370 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.097Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-03T10:34:41.370 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.097Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-03T10:34:41.370 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.097Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-03T10:34:41.370 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.097Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-03T10:34:41.370 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.098Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-03T10:34:41.370 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.098Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-03T10:34:41.371 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a[42882]: ts=2024-05-03T10:34:41.098Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-03T10:34:41.371 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:May 03 10:34:41 smithi181 podman[52068]: 2024-05-03 10:34:41.10809465 +0000 UTC m=+0.106998454 container died 906505cdab90f76f01c973b91b54cfa7941cb388f0454833584529a2c6d313cd (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-03T10:34:41.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79bfd44e-0936-11ef-bc97-c7b262605968@prometheus.a.service' 2024-05-03T10:34:41.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T10:34:41.671 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-05-03T10:34:41.672 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-03T10:34:45.138 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:44 smithi043 systemd[1]: Stopping Ceph node-exporter.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:34:45.139 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 bash[63128]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter.a" found: no such container 2024-05-03T10:34:45.402 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 podman[63141]: 2024-05-03 10:34:45.136614763 +0000 UTC m=+0.118886492 container died 695e77613650d2a7823fc511f1a0516bedcf5dcd4320af6678a202be06b387a0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-03T10:34:45.752 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 podman[63141]: 2024-05-03 10:34:45.558783213 +0000 UTC m=+0.541054949 container cleanup 695e77613650d2a7823fc511f1a0516bedcf5dcd4320af6678a202be06b387a0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-03T10:34:45.753 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 bash[63141]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a 2024-05-03T10:34:45.753 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 bash[63167]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter.a" found: no such container 2024-05-03T10:34:46.042 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:45 smithi043 systemd[1]: Stopping Ceph alertmanager.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:34:46.042 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:45 smithi043 bash[63222]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-03T10:34:46.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 podman[63155]: 2024-05-03 10:34:45.751866685 +0000 UTC m=+0.615597564 container remove 695e77613650d2a7823fc511f1a0516bedcf5dcd4320af6678a202be06b387a0 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-03T10:34:46.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-05-03T10:34:46.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-05-03T10:34:46.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 systemd[1]: Stopped Ceph node-exporter.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:34:46.043 INFO:journalctl@ceph.node-exporter.a.smithi043.stdout:May 03 10:34:45 smithi043 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@node-exporter.a.service: Consumed 4.466s CPU time. 2024-05-03T10:34:46.374 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:46 smithi043 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a[47642]: level=info ts=2024-05-03T10:34:46.077Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-03T10:34:46.374 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:46 smithi043 podman[63233]: 2024-05-03 10:34:46.098064757 +0000 UTC m=+0.111205679 container died 43fbcd25b859fb929b75218655c4ebc7df1235a5e12906546f0ef186c3e111ad (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-03T10:34:46.792 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:46 smithi043 podman[63233]: 2024-05-03 10:34:46.555410401 +0000 UTC m=+0.568551324 container cleanup 43fbcd25b859fb929b75218655c4ebc7df1235a5e12906546f0ef186c3e111ad (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-03T10:34:46.792 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:46 smithi043 bash[63233]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a 2024-05-03T10:34:46.792 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:46 smithi043 bash[63259]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-03T10:34:47.097 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:46 smithi043 podman[63247]: 2024-05-03 10:34:46.831389131 +0000 UTC m=+0.730355652 container remove 43fbcd25b859fb929b75218655c4ebc7df1235a5e12906546f0ef186c3e111ad (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-03T10:34:47.098 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:46 smithi043 podman[63247]: 2024-05-03 10:34:46.907273673 +0000 UTC m=+0.806240195 volume remove e13995d97c8c0179009c9530866b8943c9f223d444278d02f87925d681b9b63f 2024-05-03T10:34:47.098 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:46 smithi043 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-03T10:34:47.098 INFO:journalctl@ceph.alertmanager.a.smithi043.stdout:May 03 10:34:46 smithi043 systemd[1]: Stopped Ceph alertmanager.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:35:10.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-03T10:35:11.857 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:11 smithi181 systemd[1]: Stopping Ceph node-exporter.b for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:35:11.857 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:11 smithi181 bash[53151]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter.b" found: no such container 2024-05-03T10:35:12.119 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:11 smithi181 podman[53163]: 2024-05-03 10:35:11.855982917 +0000 UTC m=+0.101491118 container died dbf7b3ccb3e232157aebda6dd64e8aa33c7b0c29a2a28101b4089726fbae9c6d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-03T10:35:12.439 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:12 smithi181 podman[53163]: 2024-05-03 10:35:12.158743888 +0000 UTC m=+0.404252076 container cleanup dbf7b3ccb3e232157aebda6dd64e8aa33c7b0c29a2a28101b4089726fbae9c6d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-03T10:35:12.439 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:12 smithi181 bash[53163]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b 2024-05-03T10:35:12.439 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:12 smithi181 bash[53191]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter.b" found: no such container 2024-05-03T10:35:12.751 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:12 smithi181 podman[53177]: 2024-05-03 10:35:12.438432942 +0000 UTC m=+0.583370246 container remove dbf7b3ccb3e232157aebda6dd64e8aa33c7b0c29a2a28101b4089726fbae9c6d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-03T10:35:12.751 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:12 smithi181 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-05-03T10:35:12.751 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:12 smithi181 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-05-03T10:35:12.751 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:12 smithi181 systemd[1]: Stopped Ceph node-exporter.b for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:35:12.751 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:May 03 10:35:12 smithi181 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@node-exporter.b.service: Consumed 4.286s CPU time. 2024-05-03T10:35:12.751 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:12 smithi181 systemd[1]: Stopping Ceph grafana.a for 79bfd44e-0936-11ef-bc97-c7b262605968... 2024-05-03T10:35:12.752 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:12 smithi181 bash[53243]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana.a" found: no such container 2024-05-03T10:35:13.017 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:12 smithi181 ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a[41876]: t=2024-05-03T10:35:12+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-05-03T10:35:13.018 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:12 smithi181 podman[53256]: 2024-05-03 10:35:12.759954767 +0000 UTC m=+0.109864573 container died 15a635565ef86d7207922c482df6face44be61652c286f18b36fd8417d82fc20 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a, com.redhat.component=ubi8-container, version=8.5, io.buildah.version=1.24.2, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, description=Ceph Grafana Container, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, release=236.1648460182, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-05-03T10:35:13.369 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:13 smithi181 podman[53256]: 2024-05-03 10:35:13.116118349 +0000 UTC m=+0.466028155 container cleanup 15a635565ef86d7207922c482df6face44be61652c286f18b36fd8417d82fc20 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , version=8.5, io.openshift.expose-services=, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, com.redhat.component=ubi8-container, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vcs-type=git) 2024-05-03T10:35:13.369 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:13 smithi181 bash[53256]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a 2024-05-03T10:35:13.369 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:13 smithi181 bash[53283]: Error: no container with name or ID "ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana.a" found: no such container 2024-05-03T10:35:13.688 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:13 smithi181 podman[53271]: 2024-05-03 10:35:13.375288461 +0000 UTC m=+0.610762265 container remove 15a635565ef86d7207922c482df6face44be61652c286f18b36fd8417d82fc20 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-79bfd44e-0936-11ef-bc97-c7b262605968-grafana-a, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, name=ubi8, version=8.5, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, distribution-scope=public, vcs-type=git, vendor=Red Hat, Inc., release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , io.openshift.expose-services=) 2024-05-03T10:35:13.689 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:13 smithi181 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@grafana.a.service: Deactivated successfully. 2024-05-03T10:35:13.689 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:13 smithi181 systemd[1]: Stopped Ceph grafana.a for 79bfd44e-0936-11ef-bc97-c7b262605968. 2024-05-03T10:35:13.689 INFO:journalctl@ceph.grafana.a.smithi181.stdout:May 03 10:35:13 smithi181 systemd[1]: ceph-79bfd44e-0936-11ef-bc97-c7b262605968@grafana.a.service: Consumed 3.685s CPU time. 2024-05-03T10:35:24.736 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T10:35:24.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T10:35:24.797 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-03T10:35:24.799 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290/remote/smithi043/crash 2024-05-03T10:35:24.800 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/crash -- . 2024-05-03T10:35:24.850 INFO:teuthology.orchestra.run.smithi043.stderr:tar: /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-03T10:35:24.851 INFO:teuthology.orchestra.run.smithi043.stderr:tar: Error is not recoverable: exiting now 2024-05-03T10:35:24.852 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290/remote/smithi181/crash 2024-05-03T10:35:24.853 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/crash -- . 2024-05-03T10:35:24.898 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-03T10:35:24.898 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now 2024-05-03T10:35:24.899 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-03T10:35:24.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-03T10:35:24.930 INFO:teuthology.orchestra.run.smithi043.stdout:2024-05-03T10:21:44.631185+0000 mon.a (mon.0) 366 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-05-03T10:35:24.930 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-03T10:35:24.931 INFO:tasks.cephadm:Compressing logs... 2024-05-03T10:35:24.931 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-03T10:35:24.973 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-03T10:35:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-03T10:35:25.004 INFO:teuthology.orchestra.run.smithi043.stderr:: No such file or directory 2024-05-03T10:35:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.log 2024-05-03T10:35:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.2.log 2024-05-03T10:35:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-03T10:35:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.3.log 2024-05-03T10:35:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.1.log 2024-05-03T10:35:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.audit.log: /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mgr.y.log 2024-05-03T10:35:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.0.log 2024-05-03T10:35:25.005 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mon.c.log 2024-05-03T10:35:25.006 INFO:teuthology.orchestra.run.smithi043.stderr: 93.4% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-03T10:35:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mon.a.log 2024-05-03T10:35:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mon.c.log: 94.2% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-03T10:35:25.006 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-03T10:35:25.007 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-03T10:35:25.008 INFO:teuthology.orchestra.run.smithi181.stderr:: No such file or directory 2024-05-03T10:35:25.008 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.log 2024-05-03T10:35:25.008 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi181.kgwjoi.log 2024-05-03T10:35:25.008 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-03T10:35:25.008 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.log: /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi181.kgwjoi.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.7.log 2024-05-03T10:35:25.008 INFO:teuthology.orchestra.run.smithi181.stderr: 62.4% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi181.kgwjoi.log.gz 2024-05-03T10:35:25.008 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.5.log 2024-05-03T10:35:25.009 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.4.log 2024-05-03T10:35:25.009 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mgr.x.log 2024-05-03T10:35:25.009 INFO:teuthology.orchestra.run.smithi181.stderr: 88.5% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-03T10:35:25.009 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.6.log 2024-05-03T10:35:25.009 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-03T10:35:25.009 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-03T10:35:25.009 INFO:teuthology.orchestra.run.smithi181.stderr: 90.6% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-03T10:35:25.009 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mon.b.log 2024-05-03T10:35:25.009 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-volume.log: 81.5% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-03T10:35:25.010 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mon.b.log: 91.4% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mgr.x.log.gz 2024-05-03T10:35:25.012 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/tcmu-runner.log 2024-05-03T10:35:25.013 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.cephadm.log: 89.2% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-03T10:35:25.013 INFO:teuthology.orchestra.run.smithi181.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-03T10:35:25.022 INFO:teuthology.orchestra.run.smithi043.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-03T10:35:25.022 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-03T10:35:25.023 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/tcmu-runner.log: 66.6% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/tcmu-runner.log.gz 2024-05-03T10:35:25.023 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi043.vtftcc.log 2024-05-03T10:35:25.024 INFO:teuthology.orchestra.run.smithi181.stderr: 92.2% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-03T10:35:25.049 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-volume.log: /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi043.vtftcc.log: 62.5% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi043.vtftcc.log.gz 2024-05-03T10:35:25.050 INFO:teuthology.orchestra.run.smithi043.stderr: 92.0% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-03T10:35:25.184 INFO:teuthology.orchestra.run.smithi181.stderr: 92.4% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mon.b.log.gz 2024-05-03T10:35:25.235 INFO:teuthology.orchestra.run.smithi043.stderr: 87.4% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mgr.y.log.gz 2024-05-03T10:35:25.252 INFO:teuthology.orchestra.run.smithi043.stderr: 92.5% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mon.c.log.gz 2024-05-03T10:35:25.412 INFO:teuthology.orchestra.run.smithi181.stderr: 92.9% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.6.log.gz 2024-05-03T10:35:25.422 INFO:teuthology.orchestra.run.smithi181.stderr: 93.2% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.5.log.gz 2024-05-03T10:35:25.444 INFO:teuthology.orchestra.run.smithi181.stderr: 93.4% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.7.log.gz 2024-05-03T10:35:25.451 INFO:teuthology.orchestra.run.smithi043.stderr: 93.2% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.2.log.gz 2024-05-03T10:35:25.515 INFO:teuthology.orchestra.run.smithi043.stderr: 91.1% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-mon.a.log.gz 2024-05-03T10:35:25.544 INFO:teuthology.orchestra.run.smithi181.stderr: 93.4% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.4.log.gz 2024-05-03T10:35:25.546 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-05-03T10:35:25.546 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m0.558s 2024-05-03T10:35:25.546 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m1.983s 2024-05-03T10:35:25.546 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.102s 2024-05-03T10:35:25.560 INFO:teuthology.orchestra.run.smithi043.stderr: 93.4% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.1.log.gz 2024-05-03T10:35:25.563 INFO:teuthology.orchestra.run.smithi043.stderr: 93.3% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.0.log.gz 2024-05-03T10:35:25.606 INFO:teuthology.orchestra.run.smithi043.stderr: 93.4% -- replaced with /var/log/ceph/79bfd44e-0936-11ef-bc97-c7b262605968/ceph-osd.3.log.gz 2024-05-03T10:35:25.608 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-05-03T10:35:25.608 INFO:teuthology.orchestra.run.smithi043.stderr:real 0m0.622s 2024-05-03T10:35:25.609 INFO:teuthology.orchestra.run.smithi043.stderr:user 0m3.145s 2024-05-03T10:35:25.609 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m0.118s 2024-05-03T10:35:25.609 INFO:tasks.cephadm:Archiving logs... 2024-05-03T10:35:25.609 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290/remote/smithi043/log 2024-05-03T10:35:25.610 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-03T10:35:26.090 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290/remote/smithi181/log 2024-05-03T10:35:26.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-03T10:35:26.411 INFO:tasks.cephadm:Removing cluster... 2024-05-03T10:35:26.411 DEBUG:teuthology.orchestra.run.smithi043:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 --force 2024-05-03T10:35:26.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 --force 2024-05-03T10:35:26.968 INFO:tasks.cephadm:Removing cephadm ... 2024-05-03T10:35:26.969 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-03T10:35:26.984 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-03T10:35:26.999 INFO:tasks.cephadm:Teardown complete 2024-05-03T10:35:27.000 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-03T10:35:27.010 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-03T10:35:27.010 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T10:35:27.027 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T10:35:27.039 INFO:teuthology.orchestra.run.smithi043.stderr:bash: line 1: ntpq: command not found 2024-05-03T10:35:27.042 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T10:35:27.043 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-05-03T10:35:27.043 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv01.front.sepia.ceph.com 3 6 377 44 +424us[ +434us] +/- 49ms 2024-05-03T10:35:27.043 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 44 +414us[ +414us] +/- 59ms 2024-05-03T10:35:27.043 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -1114us[-1105us] +/- 65ms 2024-05-03T10:35:27.043 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:35:27.053 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-05-03T10:35:27.056 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T10:35:27.057 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-05-03T10:35:27.057 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 +441us[ +460us] +/- 49ms 2024-05-03T10:35:27.057 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 48 +445us[ +463us] +/- 59ms 2024-05-03T10:35:27.057 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -1095us[-1076us] +/- 65ms 2024-05-03T10:35:27.057 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T10:35:27.058 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-03T10:35:27.067 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-03T10:35:27.069 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-03T10:35:27.127 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-03T10:35:27.195 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-03T10:35:27.257 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-03T10:35:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T10:35:27.285 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-03T10:35:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T10:35:27.311 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-03T10:35:27.321 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-03T10:35:27.330 INFO:teuthology.task.internal:Duration was 1258.417088 seconds 2024-05-03T10:35:27.331 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-03T10:35:27.417 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-03T10:35:27.417 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-03T10:35:27.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-03T10:35:27.456 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T10:35:27.457 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T10:35:27.826 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-03T10:35:27.826 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-05-03T10:35:27.826 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-03T10:35:27.886 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-05-03T10:35:27.887 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-03T10:35:27.915 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-03T10:35:27.915 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-03T10:35:27.929 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-03T10:35:27.980 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-03T10:35:27.981 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-03T10:35:27.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-03T10:35:28.336 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-03T10:35:28.346 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-03T10:35:28.346 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-03T10:35:28.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-03T10:35:28.394 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-03T10:35:28.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-03T10:35:28.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-03T10:35:28.433 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-05-03T10:35:28.456 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-05-03T10:35:28.478 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-03T10:35:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T10:35:28.524 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-03T10:35:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T10:35:28.538 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-03T10:35:28.548 INFO:teuthology.task.internal:Transferring archived files... 2024-05-03T10:35:28.549 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290/remote/smithi043 2024-05-03T10:35:28.549 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-03T10:35:28.605 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290/remote/smithi181 2024-05-03T10:35:28.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-03T10:35:28.643 INFO:teuthology.task.internal:Removing archive directory... 2024-05-03T10:35:28.643 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-03T10:35:28.646 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-03T10:35:28.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-03T10:35:28.701 INFO:teuthology.task.internal:Not uploading archives. 2024-05-03T10:35:28.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-03T10:35:28.806 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-03T10:35:28.807 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-03T10:35:28.809 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-03T10:35:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 3 10:35 /home/ubuntu/cephtest 2024-05-03T10:35:28.823 INFO:teuthology.orchestra.run.smithi181.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 3 10:35 /home/ubuntu/cephtest 2024-05-03T10:35:28.824 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-03T10:35:28.941 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-03T10:35:29.047 INFO:teuthology.nuke:Checking targets against current locks 2024-05-03T10:35:29.075 DEBUG:teuthology.nuke:shortname: smithi043 2024-05-03T10:35:29.076 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T10:35:29.099 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 10:09:19.775635', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLOkh42FXMbxs2iYGdvedXuJFylnSWyEHsKB+Ia9rhtT+hfknqm4PPcd39iLIHdV5Qjxo7slCW2ubNbAo715w7U='} 2024-05-03T10:35:29.100 DEBUG:teuthology.nuke:shortname: smithi181 2024-05-03T10:35:29.100 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T10:35:29.126 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-05-03_06:31:58-orch:cephadm-wip-guits-testing-2024-05-02-0628-distro-default-smithi/7687290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 10:09:19.774458', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOEH7A8vm25PdYEe27lstXC5VT3fC2iAWSBCF7kT7dyraSFyWG8nz+a3YJ6nftkTcPWQfxlDAaq9QNTT16xYlE='} 2024-05-03T10:35:29.152 INFO:teuthology.orchestra.console.smithi043:Power off 2024-05-03T10:35:29.153 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-03T10:35:29.179 DEBUG:teuthology.orchestra.console.smithi043:power off output: Chassis Power Control: Down/Off 2024-05-03T10:35:29.179 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T10:35:29.193 INFO:teuthology.orchestra.console.smithi181:Power off 2024-05-03T10:35:29.193 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-03T10:35:29.213 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-05-03T10:35:29.217 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off 2024-05-03T10:35:29.217 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T10:35:29.239 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-05-03T10:35:33.214 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T10:35:33.240 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T10:35:33.325 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-05-03T10:35:33.349 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-05-03T10:35:37.330 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T10:35:37.350 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T10:35:37.440 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is off 2024-05-03T10:35:37.459 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2024-05-03T10:35:37.542 INFO:teuthology.orchestra.console.smithi043:Power off completed 2024-05-03T10:35:37.560 INFO:teuthology.orchestra.console.smithi181:Power off completed 2024-05-03T10:35:37.665 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} duration: 1258.4170880317688 failure_reason: 'Command failed on smithi043 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79bfd44e-0936-11ef-bc97-c7b262605968 -e sha1=f8e6d2d721bc96a93915917c59bc3d287d57263d -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | length == 2\''"\''"\''\''''' owner: scheduled_gabrioux@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6cb1cf95d415457d8bcd7005e5a5e450 status: fail success: false 2024-05-03T10:35:37.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-03T10:35:37.738 INFO:teuthology.run:FAIL